I posted to HN, and then my website was flooded with AI bot traffic

ben_w

It's too late to comment on your older posts, but I'd just like to say: it's nice to see someone making oldschool web browsers work properly on the modern web. https://www.sammystraus.com/#using-llms-to-make-the-emacs-we...

vanschelven

I _also_ posted to HN and made it to #1[0]; the effect this had on my CPU/network traffic on a 5/EUR month VPS over at Hetzner was in the single digit percentages. As per the wiki page for the C10k problem[1] where it says "_was_ [emphasis mine] the problem of optimizing computer networking stacks to handle a large number of clients at the same time". This was solved over a decade ago... Now get off my lawn! (TBF I think the author acknowledges this) [0] https://news.ycombinator.com/item?id=43725815 [1] https://en.wikipedia.org/wiki/C10k_problem

bix6

> one specific user. I won't name names Who is?!

spiderfarmer

Try posting something viral on Mastodon. The logs clearly show that it isn't the most efficient social network.

oneeyedpigeon

What confused me most about this is that the entire site is, apparently, a single page (there's no sitemap that I can use to verify this; there IS an RSS file which appears to confirm it) containing all posts. Links at the top (headed "Table of Contents") are all in-page links. At first, I assumed this was a single blog post, especially with a "Table of Contents" at the top. I guess it's following some old-school blogging conventions, but AFAICR, even those sites retained individual pages for separate posts.

switz

There is so much spam, abuse, and aggressive scraping that I've had to block: - AWS - Google Cloud - Azure - Alibaba Cloud - Oracle and a handful of other ASNs on almost all of my properties. It would be one thing if it was simple scraping, but it's often genuine agentic abuse (just spamming for vulnerabilities* or scraping millions of pages with very little valuable or dynamic content). These clouds must get the abuse happening under their own roof under control. I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time. Even Apple has aggressively ramped up its scraping in the past few months, but thankfully they do respect robots.txt. It's frustrating because some of my real users route through these clouds via their work VPNs. Also, there are a lot of less-than-reputable AI scrapers using global bot-nets via residential IPs to scrape, making it even harder to detect and ban them without affecting real people. * like leaked .env routes, brute forcing admin panels, and so on

paaloeye

Have you tried dealing with bots using CF Turnstile? Ref: https://www.cloudflare.com/products/turnstile/

ChrisMarshallNY

Heh. I posted a GitHub project Show HN, a couple of days ago, and it never even made it out of new, but it got me some bots. That's a big reason that I never post any of my apps here. Welcome to hell, kid...

janalsncm

Ok, on the one hand to have one bot hitting your website to populate some kind of slop newsletter is pretty annoying. On the other hand, “multiple times per minute” doesn’t seem that high? Even if it was 60x per min that’s fully within the limits of what a 2015 laptop in the closet can handle.

mococa

Same with me, every-time I got 1st page I got a lot of robots

Havoc

> I won't name names, but I looked them up, and they describe themselves as a service that listens for changes to a website, and then summarizes them into an AI slop news letter. And by 'listens to changes' I mean query our website many many times per-minute. Blocked. That sort of behavior would justify a name and shame imo Inconsiderate shitty services don’t deserve the politeness of protecting their name

firefoxd

Last year, I wrote a post about how I use zipbombs to deal with bot traffic [0]. It went mega viral over the web, and I regret it. My server was completely overwhelmed with bot traffic. After months of dealing with it, I had to find a better strategy then keep my mouth shut. [0]: https://news.ycombinator.com/item?id=43826798

beybol

In server logs you can distinguish the type of LLM's traffic. OAI-SearchBot/GPTBot/ClaudeBot is just indexing but ChatGPT-User/Claude-User etc. may be valuable traffic generated by user's conversation with LLM's. So it's worth choosing which ones you block, instead of blocking all of them.

andai

Interesting, thanks for sharing. >So I took a look at some data. I added top-level logging for every endpoint (how many times was a request made to each url path), and I found that a majority of the requests did not even make an attempt to fetch news articles! For reference, my website is built as a single-page-application, and what this means is that a majority of the requests did not even run the javascript necessary to render the page (!). So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over? >many, many requests per minute? How many was it? Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time? But hey. Silver lining: free load testing! ;)

andai

I'm trying to understand the reasoning behind the entity that was sending presumably hundreds or thousands of requests per minute to the same page. It's some kind of AI news site, so it makes sense they want the freshest content, but it sounds like they were doing many requests per second, so that can't be the reason?

nubinetwork

I get llm-like traffic on domains with zero web content (never was, probably never will be any). Just merely existing will get you scanned these days.

pmlnr

I really recommend blocking all traffic from anything on https://github.com/borestad/blocklist-abuseipdb

Jaxan

Oh, as soon as you add tls to your subdomain, the bots come. It’s instant.

8by3

I rolled my own version of Anubis for reasons*, and > 80% of all traffic is bots that fail the challenge, even when white listing legit crawlers. * I wanted an elixir version so I could run it alongside my existing apps instead of having it sit as a proxy in front of them. This way my existing apps can validate if the caller is already whitelisted and if not redirect to the challenger https://sphinx.8by3.com which sends you back to a site if you pass eg. https://tuhat.net

naet

I posted a link to an online game I made in the comments here once, and immediately got all kinds of good and bad traffic. Some people liked my work, found me on LinkedIn and sent a connection with a positive message. Other people tried every which way to SQL inject or attack the site, posted slurs in every available input, etc. Someone copied the site and hosted their own version with a couple minor changes. I only found that one because they had left one of the links in, which gave me referring traffic from their copy. All that from one HN comment, very much pre-AI too, not to mention the standard daily slew of requests probing for what I assume are known vulnerabilities in Wordpress or PHP. The internet is a floodgate of different actors, many of which are automated, and a link posted somewhere like HN opens the valve especially wide.