Who does Anubis actually stop?

satvikpendem

Exactly, it's the same as Cloudflare captchas where only certain blessed devices and browsers can seem to actually pass it. Ironically, adding Anubis accelerates the death of the open web.

lrem

Uh wait, a cookie that can be reused for a time to fetch the rest of the content? That's the exact opposite of what I'd intuitively want for this system...

jdlshore

Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely. Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis serves to make new connections more expensive, so that people will reuse a connection/cookie, and then falls back to normal means of preventing bad actors. (That mass scraping is the result of AI training companies, yes, but it’s the mass scraping that’s the problem, not the LLMs.)

cjd8

Funny, I'm working on a simple tool that pulls the atom feed of the latest patches from lore with Python, and I just slip a User-Agent header into the requests.get() and it works great.

yellow_lead

> The exact adversary Anubis targets defeats it trivially. Wrong, anubis stops mass crawling of web pages, by requiring a proof of work, which makes accessing these websites more expensive. Anubis was not built to stop individual users with llms.

novafunc

Anubis's primary goal is to prevent web scrapers from DDoSing a website. It's not meant to be an unbeatable challenge or only allow humans like Google's more privacy-invasive captchas. You do the proof of work, you get the content. Not all web scrapers are willing to do the work, which reduces the strain put on web servers. It's by no means a perfect system. It's goals in part prevent it from doing so. It tries to not be too annoying for humans, to not block real users, and not be privacy invasive.

throw0101d

I was curious about the name: > Anubis is a Web AI Firewall Utility that weighs the soul of your connection[1] using one or more challenges in order to protect upstream resources from scraper bots. * https://anubis.techaro.lol/docs/ > The Weighing of the Heart would take place in Duat (the Underworld), in which the dead were judged by Anubis, using a feather, representing Ma'at, the goddess of truth and justice responsible for maintaining order in the universe. The heart was the seat of the life-spirit (ka). Hearts heavier than the feather of Ma'at were rejected and eaten by Ammit, the Devourer of Souls. * https://en.wikipedia.org/wiki/Weighing_of_souls#Ancient_Egyp...

nitwit005

This mistakes the goal. It's not to block the scrapers, but to discourage excessive (and costly) scraping. The Anubis cookies are bound to particular IP address. The scrapers are often using a large set of IP addresses, so they'll be paying a far higher cost than this suggests.

patchtopic

perhaps the author, instead of the "I'm so clever" theoretical arguments from the client side, actually implemented Anubis on the server side and observe the results. I have set it up on a few sites being relentlessly hammered by clearly idiotic bot traffic, and it drops bot the traffic levels from insane to manageable. I don't even care if the traffic is AI or bots, if the bot has gone to the same effort as the author has the bot may even just access the site in a responsible manner and that's fine.

PunchyHamster

Well, my call on this thing being useless waste of time of everyone involved was correct. Compute wasted on AI tokens alone is probably far more than some cpu for token solving, better invest time into caching it well (or blocking agentic traffic entirely if that's your jam)

inigyou

We don't know who is doing the really dumb global scraping attack, but that's who it's meant to stop.

m463

Anubis does not stop me from browsing a site (not scraping, browsing with a human at the wheel) On the other hand, Cloudflare and the others stop me dead. "enable javascript and cookies." Ok. "your browser is too old". (I have an old OS with the newest firefox esr that supports it). sigh.

object-a

Maybe free market principles apply here: if Anubis fails to reduce scraper/bot load on servers, or blocks too many desired users, then the sites that adopt it would probably scrap it. If they’re keeping it even after all these posts, it must be stopping _some_ sort of undesirable traffic without costing too much desirable traffic.