Rate limits on GitLab.com are changing

I assume this is because of LLM scraping.

> A request that arrives with no credentials gets 60 requests per hour per IP address. One request per minute.

I noticed that recently Github.com has some kind of weird bot detection on public repos. I have a browser extension for switching User Agents for a specific legacy site, sometimes i forget to turn it off and Github will require me to login to view public repos. All of this is most likely due to mass scraping by LLMs. Welcome to the total shitification of the web.

Congrats on making the world worse with AI. All this performative data scraping and uploading and no progress at all.

I think it's because people are building agentic flows, reducing the amount of developer seats needed. It's the first step towards usage based pricing.

Providing kickbacks to the repos being scraped would be a good way to help fund open source projects and pay creators like streaming services do. Seems like they're headed in this direction - it would be a massive product differentiator over GH

I would spend thousands of dollars for gitlab in terms of: 1) better UX for admin panel, I'm not sure what I've enabled and what not. Several buttons do not disable the rest of the settings, leaving me with some doubts (e.g. if I disabled grafana, why is there a setting that talks about where/how I store?) 2) a minimal version of gitlab without all the AI

Github would hit four nines if they followed suit. no clue why the dont try

I understand why they're doing this, but the anticausative title kinda rubs me the wrong way.

I am honestly surprised they aren't going lower at this point. Gitlab must pay a fortune to bot traffic, most of which is malicious or garbage at best.

Damn, we’re even having Claude write important press releases now

> The requested URL was not found on this server. Getting that so I do not know exactly what they are doing. From the title I am guessing they are restricting or throttling if downloads exceeds some value.

If you are using LLMs to interact with sites like GitLab and GitHub, and you have the option to use a GraphQL API, you should jump on it immediately. GraphQL is absolutely terrible for human developers to interact with, but it's like Facebook could see into the future back in 2012. I cannot imagine a more perfect API surface for agents. With the REST API on GitHub, you can consume maybe 10 issue JSON blobs before your context window is blown out. With GraphQL constraining the results you can easily read hundreds in the same token budget. Additionally, the # of requests your agents need to make can be reduced in many cases since GraphQL can join across types whereas REST APIs cannot. You essentially get savings in two dimensions here. Quota and raw token volume per logical response.

Worst case, this could be the start of a paywall to learn from, contribute to, or host open source projects. Hopefully they find some kind of carve out for OSS projects while still blocking the egregious offenders.

60 requests per hour per IP if you haven't signed in... well that's unfortunately low.

60 requests an hour per IP seems low for a school or office network. A few people browsing issues and source files could use that up quite fast.

Walk back in 10...9...

> You get 429 Too Many Requests with RateLimit-* headers and a Retry-After. Wait the interval it gives you, then retry. Is there a test endpoint where one can validate the behavior of their ratelimit detection? Basically I do not want to cause excessive load on your servers just to test my implementation.

Is this going to apply just to ` https://gitlab.com/api/v4/:rest_of_the_url ` endpoints, or also to the API-ish endpoints like ` https://gitlab.com/:org/:repo/raw/HEAD/:path `?

This is why I moved my code to self hosted forgejo instance. Private and guarded behind self hosted OIDC instance. No more worrying about "rate limiting," subscription hell, or random extended outages (ie, github). If LLM wants access, might implement payment layer and use 402 http status code and redirect them to payment page ;). Wonder how many people just give agents carte blanche physical (credit card) and virtual access

Rate limits on GitLab.com are changing

Discussion Highlights (20 comments)

Related Discussions