I built a 500k-domain search engine for makers in a weekend for $10

dreamforever

Check out my latest project! You can fork it, tweak the policy manually or with AI, run the system and watch the data come in! It's engineered to keep a low data footprint, so 500k domains fits into 1GB on disk. If you have local models it's free! You just might not get the best throughput depending on your GPU. My production data is not exposed anywhere yet, and I may never expose it. The point is for you to fork and make your own policy, and thus your own personal search engine! The article covers basic analysis on my data, so it's worth a read if you're interested! A deeper analysis may arrive with V2 if I ever do it

iFire

Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. 5. `code is going up as open source` soon (TM)

headz

TS;DR: Too Sloppy; Didn't Read.

dewey

I think Kagi Small Web filter would give you very similar results.

marginalia_nu

Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea. I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.

orliesaurus

Like a personal Google? How do you bypass all the captcha, ip bans, cloudflare turnstile antibot stuff etc?

eggbrain

This is actually where I see software going in the short term -- cloud moving to local. A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search. But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it. The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.

pimlottc

Sometimes I think people forget how capable computers are. 500k is not much. You can just slap that in a Lucene instance. This is a solved problem.

elorant

Domains are way more than just 40M though.

fg137

Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.

pavel_lishin

From the screenshot, it's very funny that one of the indexed sites is www.llresearch.org, which looks like it's run by a crackpot.

BaudouinVH

How do you build a list of domains you want to index ? I see there is a fetcher and a spider in the code but so for I haven't found how to build that list.

lagrange77

Took me a few minutes to realise it's not a domain name search engine.

tpowell

It takes a bit of setup and a huge download, but every time I need a good domain I follow this old post from Derek Sivers. I have Claude de-dupe it and turn it into a searchable database (on my machine), then have it search genres and terms I'm looking for. It's a task Claude is very well-suited to, from the technical implementation to back-and-forth about selections. [link]: https://sive.rs/com

eichin

Reminds me that AltaVista's servers ran in 4G of RAM (there were famous, at the time, pics of the circuit boards - DEC was rightfully proud of this, 30 years ago) and that a modern AltaVista should run on a decent laptop :-)

NetOpWibby

This is a damn good project. Makes me want to make headway into an idea I've had for quite some time P2P search...we'll see.

gosolozero

Built a free CLI for this as well: https://github.com/solozerolabs/Namera Checks socials and trademark too

cpill

I was thinking that a search engine that ignores anything with advertising on it would be useful. This has inspired me to give it a go (on the weekend even)

frogger8

FYI for those needing a list of domains Subject: I want all domains and subdomains https://groups.google.com/g/common-crawl/c/XC2QmOE-sdI?pli=1 or google for COMMON CRAWL

deadbabe

The author struggled with categorization simply because they did not truly understand k-means clustering, a fundamental concept in this kind of computing science. You cannot just let a model run wild.