Remix.run Logo
▲ phendrenad2 3 hours ago

I did this for a while. I went so far as to built a highly advanced web crawler webapp that was better than the Internet Archive in a lot of ways.

But the modern web has made this unreasonable. Every webpage is 99% junk (CSS, JS, HTML structure, tracking pixels, UI images, navigation links, cryptominers (I joke)) and 1% content. You need a smart AI system to pick out the good parts, otherwise you'll quickly find yourself schlepping hundreds of gigs of junk to just preserve your favorite websites.

The better idea is: if you love information, back it up. Collect the knowledge from the website, not the website. And then preserve it. Create a free ebook. Or sell it on Amazon kindle. That'll last a lot longer than the website will, and any HTML backup of the site you could possibly make.

▲garbagecreator an hour ago | parent [-]

I'd suppose most of those junk websites have no value in them, really. The most valuable information in the web is in good ol' (x)html, maybe some css. But something like readability.js [1] can help in filtering the junk around the main info.

[1]: https://github.com/mozilla/readability There's also a C rewrite: https://github.com/eafer/rdrview