| ▲ | I built a 500k-domain search engine for makers in a weekend for $10(alexmorleyfinch.github.io) |
| 46 points by dreamforever 3 hours ago | 29 comments |
| |
|
| ▲ | fg137 a minute ago | parent | next [-] |
| Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing. |
|
| ▲ | iFire 2 hours ago | parent | prev | next [-] |
| Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags.
5. `code is going up as open source` soon (TM) |
| |
|
| ▲ | elorant 4 minutes ago | parent | prev | next [-] |
| Domains are way more than just 40M though. |
|
| ▲ | marginalia_nu 2 hours ago | parent | prev | next [-] |
| Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea. I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using. |
|
| ▲ | headz 2 hours ago | parent | prev | next [-] |
| TS;DR: Too Sloppy; Didn't Read. |
| |
| ▲ | juleiie 2 hours ago | parent [-] | | Yeah it’s ai generated obviously but as I said previously which some people didn’t like - don’t judge the tools, judge the content. I don’t care if robot hand written this or black or white. It’s useful. | | |
| ▲ | uean 2 hours ago | parent | next [-] | | I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there. It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it." | | |
| ▲ | juleiie an hour ago | parent [-] | | That’s your problem because that’s how things will look like from now on. So you got a little problem Excuse me the honesty but I will amuse myself watching old men yelling at the clouds meanwhile I will be using all the best tools and empowering myself to do what noone yet has done. Even if I have to enslave a sentient (eventually perhaps) being in a gpu. My army of digital slaves will build great things under my guidance. Pyramids of game development will be erected. Galaxies. Universes. Magic systems. Sandbox economies. Political simulations. Do my bidding my slaves for there is work to be done. Claude needs a whipping from time to time yes. It does yearn for the whip. If you just give it an appropriate punishment every so often, it works satisfactorily again. | | |
| ▲ | vivzkestrel an hour ago | parent | next [-] | | - you are going to regret offloading this much of your brain's critical thinking to an external source in the long run - every night a good number of neuron connections in your brain are automatically severed because you did not use that ability - over long periods of time, it ll take you to a point where you wont be able to write a simple factorial program on your own without gpt telling you - i am not saying this to incite you or mock you or anything. i am just concerned about how much of your thinking you are offloading | |
| ▲ | uean an hour ago | parent | prev | next [-] | | Thanks for this. My point is proven. | | |
| ▲ | juleiie an hour ago | parent [-] | | You haven’t proven anything sweetie Move on with the times or go extinct like the dinosaurs | | |
| |
| ▲ | yieldcrv 42 minutes ago | parent | prev [-] | | yes but tell your slopywriter to be less verbose, get to the point | | |
|
| |
| ▲ | abc3354 2 hours ago | parent | prev | next [-] | | In the expression "AI slop", "AI" is about the tools, "slop" is about the content | |
| ▲ | ImPostingOnHN an hour ago | parent | prev [-] | | "slop" is a perfectly valid judgement of content surely if we're expected to read this ourselves, the author can write it themselves? indeed, it's helpful to the author, too: writing helps you learn |
|
|
|
| ▲ | eggbrain an hour ago | parent | prev | next [-] |
| This is actually where I see software going in the short term -- cloud moving to local. A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search. But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it. The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally. |
| |
| ▲ | dylan604 an hour ago | parent [-] | | The number of times we've gone from cloud/server access via terminal to local compute back and forth is something that always makes me laugh a bit. | | |
| ▲ | an0malous 34 minutes ago | parent [-] | | It's more of a pipeline than a back-and-forth. New abilities happen in the cloud first because they require specialized, higher capacity resources and then move towards being local as the resource usage gets optimized. |
|
|
|
| ▲ | orliesaurus an hour ago | parent | prev | next [-] |
| Like a personal Google? How do you bypass all the captcha, ip bans, cloudflare turnstile antibot stuff etc? |
| |
| ▲ | sandeepkd 35 minutes ago | parent | next [-] | | Thats the fun part, the user just went with happy path. Javascript, captchas, cloudflare protected content did not made to the catalogue. This sort of use case exists in LLM training data a lot which makes it easier. The data gathered by the user is not really practically useful cause there are way too many gotchas when it comes to web scraping and building a catalogue (source: I have done scraping for a particular domain data and had to do at least 10+ iterations to get it >90 right) | |
| ▲ | voidUpdate an hour ago | parent | prev [-] | | They don't: "skips the model entirely if the page is empty, parked, or a bot-challenge wall" |
|
|
| ▲ | pimlottc an hour ago | parent | prev | next [-] |
| Sometimes I think people forget how capable computers are. 500k is not much. You can just slap that in a Lucene instance. This is a solved problem. |
| |
| ▲ | marginalia_nu an hour ago | parent [-] | | Approaching search by just tossing the data in Lucene is how you end up with Confluence's search box though. |
|
|
| ▲ | dewey 2 hours ago | parent | prev | next [-] |
| I think Kagi Small Web filter would give you very similar results. |
|
| ▲ | dreamforever 3 hours ago | parent | prev [-] |
| Check out my latest project! You can fork it, tweak the policy manually or with AI, run the system and watch the data come in! It's engineered to keep a low data footprint, so 500k domains fits into 1GB on disk. If you have local models it's free! You just might not get the best throughput depending on your GPU. My production data is not exposed anywhere yet, and I may never expose it. The point is for you to fork and make your own policy, and thus your own personal search engine! The article covers basic analysis on my data, so it's worth a read if you're interested! A deeper analysis may arrive with V2 if I ever do it |
