Just the rumour of a bug is enough to find an exploit these days

nickcw

This describes my life as an open source maintainer at the moment! In the first 10 years of the rclone project we received about 20 security disclosures through GitHub. We had to deal with over 40 in the last month! That has taken a huge amount of my time, even using AI tools to triage and come up with fixes for review. The hit rate for those security disclosures is pretty good - about 75% of them have a nugget of something which needs looking at. The configurations for rclone have got increasingly unlikely so I'm hoping they will dry up eventually. I was considering just merging the fixes straight to master just to make my life easier rather than holding a dozen independent security fixes on branches and merging them at the point release and hoping not to have too many conflicts to fix up. I've decided to stick with the process for the moment. GitHub assigns CVEs for the advisories. Before the AI apocalypse they took 2-3 days for an assignment but now it they are running at 3-4 weeks so I have to send the point releases out with CVE-PENDING in the changelog which isn't ideal. Not sure what the solution is, but it is definitely a problem for us.

bri3d

I don't think this is new with LLMs (finding an exploit based on a few words offhand has always been a fun part of exploit development), but it's scaled and democratized to mass exploitation of low value targets. Backing exploit PoCs out of patches, commit messages, and random overheard or over-read sentences is a practice as old as vulnerability research. The difference with LLMs is that an explosion in actors "skilled enough" (human or not) has enabled sloppy / low-skill "exploit the whole Internet" actors in a way they weren't previously enabled. I do agree with the author's ideas, though; most of these are things that should have been done much sooner, and I suppose it's good in a sense that there is a forcing factor now.

godelski

It's easier to find bugs, fix them, yet there's less will than ever. My bosses just want speed and will give me a 30 minute lecture on why I don't need to solve a bug that Claude solved in 5 minutes, I've verified, and it's already in an open PR. All the while we're pushing out bugs faster and faster. No matter how good AI gets at fixing bugs we'll never fix them when there's no will to fix things. Software will never be good if there's no will to make good software. The problem has always been about will. To many better products. It's insane that in a time where we can do better on speed and quality we still choose speed and tell ourselves it's velocity

loeg

No mention of memory safe languages? Sure, it does not help existing projects, and sure, of course you can still have logic bugs (or memory bugs using escape hatches like unsafe). But they do help significantly in reducing the number of exploitable bugs.

zingababba

Transferring ideas in their abstract form is entirely possible now. What this blog post describes I believe even undersells the current reality. I've had success with finding crits by pasting in extremely high level or abstract concepts. I recently went from a talk abstract (3-4 sentences) to crits lol (finding that vuln type).

Tomte

So we have automated Thomas Pornin?

petesergeant

Even that sounds too involved, just tell the agent that one exists, even if it doesn’t… https://sgnt.ai/p/terrible-mistake/

saghm

Sounds like the next step is to start rumors about vulnerabilities so that someone else will find them and you have plausible deniability. "Hey, did you hear? Someone told me <competitor company> might be leaking their logs from a publicly accessible endpoint..."

stephbook

I think roll-out and deployment are even bigger issues. Who updates their software stack within 10 minutes? Most CI runs take longer to verify the business logic is still working. Add to that the danger of supply-chain attacks where you don't even want automatic updates.

jameshart

I do wonder what the hit rate is in general for Claude finding a successful exploit when prompted with a rumor that leads it to assume the bug is there. “I’m told there’s a path traversal exploit in this package. Can you find it?” - probably a reasonably high chance of it finding one, even if you just made that rumor up.

xbar

Hm. I wonder if you can force an exploit into existence with a false rumor of a bug?

westoque

with agent loops, the cost of exposing bugs should be cheap now. every pen tester should be using agent loops and on the flip side, an agent could easily see this issue once filed and a fix immediately done. it's incredible how cheap code is now.

ChrisMarshallNY

Sadly, it seems that the lesson from this, is to keep your repos private. I am not a fan of that, but I think many folks will take that away from this.

29349175

"This particular report arrived privately on a Slack channel via Jane Street last week, and was itself found via Claude Fable. That compresses all timelines considerably..." So Fable knew about it. Maybe someone is running experiments again like in OpenAI's Huggingface hack. Cute to see that the Glasswing apparatchiks still protect their income stream and hand out no accesse.

rndhouse

I built a tool that monitors commits and tries to detect silent bug fixes. With GPT-5.5-class models, it can identify fixes hidden within otherwise routine commits quite reliably. Obfuscating the code changes enough to avoid detection is difficult. I have heard of at least one project (c-lightning?) temporarily releasing a closed-source binary as a workaround until users could update safely.

metrofun

proves you only need conviction to break smth

janpeuker

I do get the anxiety about new bugs but to be honest I fear more that in a few years it's so cheap to fix medium to high security bugs that lowkey hacking and finding worrying privacy violations becomes prohibitively expensive for citizens.