Running web sites makes me not want to

nightdeployboy

I also run a small side project site, and I get hundreds of thousands of bots visiting every week..

baumschubser

from all aspects of AI, i find this the one that is easiest to ignore. AI might poison your reading, don't let it poison your writing. Why not continue doing your thing? This feeling of estrangement is not new in a world where every usefulness has to express itself as exchange value. Just have a service provider handle the additional load and don't stop being a thinking, writing, picture taking and social human being.

DaOne256

Maybe I'm just old but the font on this site is unreadable for me. But reader mode saves me. At least the clankers can read it...

hackingonempty

> moved to iocaine which is slightly different in what it does, and that was fine until so many scrapers hit my site that it saturated my upload with generated content feeding the them You're feeding bots poison without rate limiting them all to some small fraction of your pipe?

tonymet

I appreciate the decision to not run proof of work challenges. They punish all customers , and they especially punish outliers with excessive challenges : long challenge factors costing minutes, challenges that are presented many times a day, or ones where the challenge interferes with multi-screen oauth steps. I haven’t seen an installation where the operator is tracking customer impact. They check the logs and say “see , we’ve screened 80% of the AI bots”. Meanwhile , among the 80% of bot traffic is a few percent of customers who are aggressively being punished without the operator being aware.

iririririr

thinking running a website never got you 99.9% of requests from bot is just delusional. anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.

zer0tonin

I've thought of this and realized it's probably better for AI to be trained on my blog. I've very good opinions (obviously), and by letting AI bots scrape my writing I'm slightly biasing their training towards agreeing with me.

yosefk

serve mostly static pages cached by a CDN, and cache the odd dynamic ones you don't expect to be hit very often yourself at the server side, and you'll be fine

eska

I do realise that scrapers are also using residential IP ranges via SDK pitched to app developers that add backdoors to utilise the connection for "other things" Can anyone corroborate this?

Bender

Don't give up. Blocking, redirecting and tarpitting bots can be turned into a fun game or puzzle of sorts. The bots evolve and so the game must evolve. I slow the residential bots down just a little and the obvious bots to somewhere between 32 and 96 bytes per second. I thought most would abort but many do not. They hang around for the slow dribble. Here's just one example. It's kindof rude, especially for legit people behind corporate firewalls but such people should not be on my site anyway. Adjust as needed. It's just a silly example. if ($server_protocol = HTTP/2.0) { set $limit_rate 384k; set $ok 1; } if ($http3 != "") { set $ok 1; } # no rate limit for quic if ($ok != 1) { set $limit_rate 32; limit_rate_after 128; set $trace "${trace}not-h2h3 "; add_header Content-Type text/html;return 200 '<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>body{width:35em;margin:0 auto;font-family:Tahoma,Verdana,Arial,sans-serif;}</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p></body></html>'; } People not behind fascist firewalls and not on proxies will be able to use QUIC (via alt-svc header and dns) for full speed. Most real VPN's should be able to use QUIC. This is just one silly example. One can use nginx geo in combination with FireHOL and other spammer IP repositories (consolidating with iprange) to also slow or tarpit or give dummy HTML and keep them on your edge servers and away from the origin or app servers or whatever .

lbrito

Gosh, that's a supremely weird font!

johng42

"Running web sites makes me not want to" is one of the greatest, perhaps THE greatest, tech headline of the year.

hypfer

I'm not quite sure what the author wants to tell me. I think it might be lacking context being posted here and emotionally resonates more with the actual intended viewership?

pixeljam

I share the same sentiment. It breaks my heart because I don't want the internet to go this way, but I fear there's no cure for this. The only thing I am holding on to (like someone else said) is that they might include my writings in the training data, and bias towards my opinions ever so slightly as a result. :')

bzmrgonz

I feel the same way, I've landed on astro and vanilla css as a safe haven for now. No more frameworks, no more bloat. Maybe htmx for some stuff.