Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister (https://hister.org/).

Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews and the full page content via MCP.

Take a look at how the MCP can be utilized: https://hister.org/posts/give-your-ai-assistant-a-private-me...

▲

zeroq an hour ago | parent | next [-]

I'm sorry for not taking the time to read the docs, but I have a question.

Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name.

For years I've been collecting links to different work related trivia which I use on a daily basis as a rabbit-from-a-hat solution to answer random question from friends and coworkers. For example someone randomly asked me for an idea for color palette for data charts and I can immediately give them a scientific research into the color palette. Or an obscure algorithm.

But with time the collection has grown substantially and it's really cumbersome to find the proper things.

Would your project be a good fit for my problem?

▲

asciimoo an hour ago | parent | next [-]

Absolutely, this is a great example where Hister can shine.

I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.

	▲	zeroq 31 minutes ago \| parent [-]
		would it be possible to gdrive/rsync/git the data between machines and then use the data on an online server for retrieval (given that I would handle data sync myself)? also what exactly are you using for search? does it support trigrams? how do you sort results?

▲

justusthane an hour ago | parent | prev [-]

Also very interested in this. I was playing around with doing the same thing with YaCY. I want the proxy aspect so that I can proxy my phone traffic through it as well.

	▲	asciimoo an hour ago \| parent [-]
		Unfortunately mobile Chrome browsers don't support browser extensions, but our extension works well on mobile Firefox.

▲

derrida 23 minutes ago | parent | prev | next [-]

Wow! that looks like a bit of software I have been dreaming about for awhile - will definately check out! You're at least doing something right in communicating the reasons why and appeal for starters! All the best!

▲

blackqueeriroh 19 minutes ago | parent | prev | next [-]

Oh thank god there used to be several tools like this and they slowly went away. I’ve been wanting this to return.

▲

MrDrMcCoy an hour ago | parent | prev | next [-]

Always excited to see new things like Hister in the search space. What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful? Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?

	▲	asciimoo 36 minutes ago \| parent [-]
		> What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful? There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation. > Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index? My long term goal is exactly this. I'd like to add federation/P2P feature [2][3] to evolve from being a private search companion. I'd appreciate any help designing the system. [1] https://blevesearch.com/docs/Home/ [2] https://github.com/asciimoo/hister/discussions/432 [3] https://hister.org/posts/public-search

▲

chrisss395 an hour ago | parent | prev | next [-]

I love your idea and wondered why saving and indexing browser visited pages was not being done. Does this handle large amounts of local files, for example 10-20TB across file types like Powerpoint, Excel, Word, and PDF?

	▲	asciimoo an hour ago \| parent [-]
		In its current form it cannot handle this amount of data efficiently (and doesn't support powerpoint/excel/word yet), but this is a valid use-case, I've added a TODO item to experiment with it.

▲

2 hours ago | parent | prev | next [-]

[deleted]

▲

kristianpaul 2 hours ago | parent | prev | next [-]

Is this similar to fastcrw ?

	▲	asciimoo 2 hours ago \| parent [-]
		Both are search engines, but that's all the similarity. Hister has a traditional crawler, but its biggest strength is automatically indexing browser tabs as those are rendered. This way it bypasses authentication, CloudFlare, captchas and most of the annoying limitations of traditional crawlers. Hister also provides full offline result previews. Check out the small read-only demo: https://demo.hister.org/

▲

operatingthetan 3 hours ago | parent | prev [-]

I installed this a while back and honestly I almost never touch it. It turns out that for me searching my history doesn't really replace a search engine at all. The built in extractor list is pretty limited and adding them seems like too much of an ordeal for me to bother.

	▲	asciimoo an hour ago \| parent [-]
		Sure, it cannot fully replace web search engines (yet), but it can reduce the dependence on these services more and more as your index grows. Hister is designed to support quickly falling back to traditional search engines with a single hotkey if no results found. I agree, we should add more extractors [1]. Can you recommend extractors you missed? [1] https://github.com/asciimoo/hister/issues/305