Cloudflare CEO is lying to you about the bot traffic jump

kordlessagain

I concur and have been talking about this for a while. The fact is, Cloudflare is a man-in-the-middle. That's their focus, that's their purpose. They will limit your local crawler from accessing pages. They will demand you use their crawler. They will decrypt your traffic if they get a warrant. They always decrypt your traffic anyway, but they will give it to state actors if they demand it. That's not to say anyone should break the laws, but the issue right now is that intellectual property is incompatible with what is coming with AI. I don't hate on Cloudflare because it's a bad service. It's actually pretty good, but the fundamental problem is they make their purpose to be a single choke point of all data on the Web. That's not right. It never was.

taeric

I confess a sad assumption that bot traffic is far higher than we have admitted for a long time. Though, maybe we would see different stats specifically to social media sights to astroturf like counts? Certainly feels that we have known for a long time that bots were larger in ad viewing than ad companies wanted to admit.

1vuio0pswjnm7

There is a unfortunate incentive created when a "business" (MiTM) depends on "bot traffic", i.e., the continued nuisance of bot traffic, to make money If the "bot traffic" declines, then the "bot protection business" goes down with it Cloudflare communication are sometimes careful to refer to traffic _labeled as_ bot traffic versus actual bot traffic Because the "business" relies on the existance of "bot traffic", theres an incentive to broaden the scope of what is labeled as "bot traffic" The false positive rate can be high. The public should see those statistics, and in truth it may be infeasible to compile them when theres no verification and the entire system relies on heuristics "Bot protection" can be used to gather fingerprints for marketing It can be used to force users to use certain software, e.g., certain browsers, and to enable Javascript subjecting users to data collection, surveillance and ads Originally the motivation for avoiding "bot traffic" was based on behaviour, e.g., exceeding acceptable rates of usage, making too many requests in a given time period, exceeding rate limits Now it's available to exclude traffic based on criteria such as what browser someone is using. NB. This is more than "user-agent string". The company forces people to sign NDAs before telling them what it is doing to fingerprint www users If residential proxies are the problem then why not go after the companies that provide them The truth is that those companies are not the problem. Their customers are so-called "tech" companies Perhaps it's these so-called "tech" companies that are the problem Certainly the problem is not the individual www user who doesnt use an "approved" graphical, Javascript-enabled browser who gets blocked or fingerprinted trying to make a single request But thats who suffers from "bot protection" so that so-called "tech" companies can profit from data collection, surveillance and ads

reconnecting

Cloudflare bot detection has taught me a reflex to close the tab every time I see its logo.

mikey_p

Do people really expect CEOs to be knowledgeable about any technically details in 2026? My experience is that CEOs are getting increasingly out of touch with what their employees actually do and what their customers want.

simonw

"Cloudflare CEO is lying" is a bit of an aggressive take when he linked to the exact data so you can see it for yourself - and that's how this article was able to analyze it: https://radar.cloudflare.com/traffic#bot-vs-human Update: I see the problem. Here's the full tweet: https://x.com/eastdakota/status/2062212701414187452 "Thought it would be end of 2027, then early 2027, but agentic traffic growing so fast that bots have now passed human traffic online for the first time in the Internet's history." But the quoted segment in the article was just "…bots passed human traffic online for the first time in the Internet’s history." It looks to me like the data supports "bots passed human traffic" but does NOT support "agentic traffic", since more of that traffic is from AI crawlers building indexes than from agents that are browsing the web on behalf of their owners. If that's the point the article is trying to make then the headline is a little more supported, though I'd still say it's too hype-y a headline. I guess a lot of this rests on what you assume "agentic traffic" to mean.

bobjordan

The article strikes me as quite uncharitable to characterize it as "a lie". I doubt this CEO just sat down and calculated he was going to write lies. While it's fine to call out he's wrong per his own fuller data set, it's quite a different thing than calling the person out as a "liar" in a rage-bait fashion.

DevKoala

Not sure if the Cloudflare CEO is lying, but I have a pixel deployed on tens of thousands of sites offering B2B solutions, and bot traffic overtook human traffic this year.

thm

Why did we start treating Cloudflare (a public, for-profit company) as the undisputed authority on anything related to the network layer of the internet in the first place?

Bender

I tested this theory not long ago and did not see anything that aligned with the hype around bots. [1] There are indeed more bots than humans because of course there are or at least the appearance of . Bots crawl everything linked from popular sites whereas humans only click on things that interest them and even then they do not typically siphon the entire site. There are new bot operators every day due to curiosity and FOMO. The only thing I saw that could possibly be construed as abusive were some poorly configured RSS bots. Even when my server told the bot that the page would not change for 4 hours the RSS bots would check every 10 minutes meaning they are ignoring the cache-control header. This was entirely harmless, just slightly annoying. The RSS bots are not new. Most of the bots are not even trying to disguise themselves as humans. Most bots are not programmed to parse cache-controls, rel tags or fetch robots.txt meaning they only follow the pirate code. A bot will do what a bot can do . I was expecting the bots to mirror a couple git repositories I exposed but they did not go deeper than the README.md. None of them. I think this is the same pattern of catastrophization that exists around AI dooming the world and I don't know why it is spreading. I guess it must work or people would not do it. [1] - https://blawg.nochan.net/b/Internet-Crap/20260522-Maybe-AI-B...

tyjkot

I love when smart people catch liars.

ChrisArchitect

Discussion: https://news.ycombinator.com/item?id=48387144

nothrows

Cloudflare is junk. Their entire billion dollar service can't distinguish my (DAILY) GET request to mainstream news sites from bot traffic, nothing they say or do is of any value. I've had the same IP for decades.

csomar

The article is a bit too strong, aggressive I’d even say. Content is loaded only if the bot executes JavaScript and loads all content willingly. These do exist, but they are more expensive to run than a basic curl bot. It’d make sense as you might not want your bot to load everything a real human would do (ie: analytics, ads, unrelated files, etc..) and only focus on the content. Also, am I the only one surprised that bot traffic is not the majority already? For my site, it’s x100 bots for every human.

jimrandomh

I deal with scrapers that sometimes border on DDoSes for LessWrong. The amount of bot traffic varies greatly between sites; if you have more URLs you get more bot traffic (regardless of whether those URLs represent a deep content catalog, or useless URL parameter permutations). It's bad for LW because of the content-catalog depth. It's easy to drastically underestimate the amount of bot traffic, because bots make efforts (of varying sophistication) to look human enough to evade blocking. That includes using fake user-agent strings corresponding to real browsers (often but not always with implausibly old version numbers), proxying through residential IPs, and sometimes using full headless browsers. In my own data, traffic from badly behaved browser-impersonation bots exceeds traffic from named scrapers like GPTBot by something like 10x. The measured percentage of bot traffic is higher for HTML than for other content types because many bots will load an HTML page, and then not load the JS/CSS/image/etc resources it references. But these are the least-sophisticated and most-detectable bots.

eli

"Lying" is not supported by the evidence. In the context of bot traffic on the web, looking at only GETs for HTML is a reasonable approach. If you're counting all requests for all assets then a single page view of nytimes.com would count 100x as much as one for HN. I would assume a lot of people running websites tend to think in pageviews, especially when dealing with bots because images and CSS files tend to be "cheap" static content but HTML requests are often dynamically generated. It's also a single tweet that links to the data used to "disprove" it. Would be a weird way to lie.

wiredfool

I run some moderate profile gov and ngo opendata sites, and I’d say that bot like traffic is 99% of the requests we’re seeing on some sites. Mostly current valid user agents, lots of ip addresses, but the traffic patterns are not organic. I’m not clear if it’s bad ai scraping or dos, but at some level it’s indistinguishable.

cheeseblubber

Anecdotally the site we manage we are easily seeing 100x the traffic from bots than from humans in the past year. So much so that it is impacting our hosting costs.

NetOpWibby

The amount of people in these comments demonizing Cloudflare services or conflating their very existence with rubbish/trash/nonsense because you personally know better is wild. Y'know what would be better? Making a site showcasing what tools are better than Cloudflare's services! And how to use them! And sharing said site so people know about them!

dentemple

A lot of folks in these comments complaining about Cloudflare, but not many suggesting alternative solutions.