An update on residential proxies and the scraper situation

cyanydeez

mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS. The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

tingletech

The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org . I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?

atomic128

There is a large community of people that poison scrapers. The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc. Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

mips_avatar

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

everfrustrated

I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.

sixtyj

The issue with scrapping is the intensity and volume of bots. I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it. Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.

Bratmon

Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.

eduction

Can BitTorrent’s architecture contribute anything useful here? I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.

tiahura

Again, why do we allow China on the Internet? Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.

dang

One article mentioned in the OP was discussed here: Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)

andai

>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time. I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks. Then they'd switch to IPv6, and... well, are we using IPv6 for anything important? Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)

zb3

> widespread scraping of web sites in search of training data for large language models and related projects This is a good thing, thanks to this we have powerful open source LLMs. > This activity overwhelms sites with traffic. When LLMs get good enough, we won't need those sites anymore :) [not satire, this is what I think, without self-censorship]

arjie

What a pity. Mostly I just want personal archives of things so that I can search them much faster than commercial solutions and the like.

harshreality

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on. It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock. The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high. For websites running dynamic languages, a binary (anubis is in go) sentry that operates before [1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha. The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block , or they have to use massive amounts of compute. [1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.

627467

Has no one noticed their miniflux instance failing to fetch feeds because of this?

WarOnPrivacy

https://archive.fo/PAcF5

rao-v

I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands. Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use! EDIT: Lwn.net is perhaps not a fair target of my ire. “There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.” Is reasonable! Many others are not

TZubiri

>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on. The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0. Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.

Avery29

Google itself is a huge database.Who makes these rules depends on who's leading the market.

ValentineC

From the article: > More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact. I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?