How to Block Some of the Bots

binaryturtle

410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks! Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)

fxtentacle

I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.

iririririr

most (all?) of those will 100% block valid traffic too

ajsnigrutin

There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.

Capricorn2481

Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.

RobotToaster

Blocks firefox's built in VPN.

CqtGLRGcukpy

If you are unable to read this, there is an archived copy at https://archive.ph/d3236

genodethrowaway

and all valid traffic too, judging by these HN comments (and my own attempts to connect).

FabCH

The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive. The added explanations by the author in this comment thread are hilarious. You sir are a good writer.

fatty_patty89

what's up with those response headers? "adult" ... "ai" ... response length 68

boznz

I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.

userbinator

You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago. Block on traffic volume and request frequency if that's causing a problem. (And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)

juleiie

The guy is certainly kind of visibly ravaged by the more shady denizens of the global internet. But there are some fun things to read there in any case.

billfor

I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time. Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.

AussieWog93

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?

inigyou

I can't read this article because I'm a bot, apparently.

inigyou

Lol blocking 10/8. Just why.

thayne

You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP. Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers. > One class of bots always claims to come from the root page of google. Obviously this is a lie. This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.