Going Dark, and the era of law enforcement hacking

Carrok

Sounds like a pretty strong argument to self host, and otherwise be in charge of the software you use.

bell-cot

> Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon. > Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs. His conclusion sounds extremely optimistic to me.

tolugenius

> In this case, we’re just going to have to hope that this time we make the right choices, for no other reason than that they’re right. I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.

Gigachad

I'm supposed to be concerned that the US government and Israel won't be able to hack everyone's phones?

Insimwytim

On one side, you have pieces like this, where seemingly there are constant fights between serious actors with large and properly distributed budgets, employing top tech and top minds; on the other - regular news of the hackz, where responsible person in charge of security with root access failed to grasp basic technical knowledge (several times), ticking every checkbox in "never do this" list from security best practices, which led to every customer being pwned. It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross.

natecodes

> This is not a call to action for experts to rally behind a sophisticated plan. Like so many things about the AI revolution, it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today. heh. long greasy slide. It really does feel like that.

Scryptonite

I think that one of the reasons they (frontier companies and the gov) will be putting so much effort into curtailing bugs and vulnerabilities is to limit the blast radius of future AI models. Imagine with the new Sol Ultrafast, they could have pwned Hugging Face in 6 hours and not 4 days (IIRC). It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.

mbroshi

> In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon. This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite. My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_.

pianopatrick

I dunno man, if there's a deluge of new AI generated code at all layers of the stack I think there will still be vulnerabilities. Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base. But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.

gmuslera

No system view. The law agencies can develop exploits to intercept our phones, that is a new, and totally unseen before threat. Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones. You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.

embedding-shape

> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure. I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore. Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention? I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.

wavemode

This "going dark" scenario would require new legislation. With secure enclaves, modern smartphones can't be cracked open in the manner that the FBI wanted in the 2016 case. So there's no such thing as "court order tech company to crack phone" anymore. It would have to be "outlaw tech companies from producing phones that they can't crack open", which is very different and does not fall under any existing US statute.

BoingBoomTschak

Childish, nation-states as powerful as the US have access to much more potent stuff. Maybe they'll be forced to rely more on their Intel ME/AMD PSP/modem (cf https://redmine.replicant.us/projects/replicant/wiki/ModemIs... ) backdoor and ANT James Bond catalog.

Grombobulous

I think what’s unintentionally eye-opening about this chart is the recency of “law enforcement can read your text communications.” Law enforcement doesn’t need this surveillance ability at all. All time periods prior to 25 years ago didn’t have it. Additionally, there is no correlation between “law enforcement reads text messages” and crime rates going down.

bottlepalm

Man I thought from the title this was going to be about next-gen AI being able to zero day everything so effectively that software security is meaningless and we'd need to basically shut it all down, go dark.

jrflowers

> I’m concerned that AI is going to make software much too secure. Lmao this is like “I’m concerned the raccoons that I see in the storm drains are going to make our sewer system much too efficient”

cadamsdotcom

We will know we've made systems secure when laws focus on compelling people to provide access . These laws exist - they aren't the focus yet. Right now there's still no need; just hack the device or compel the cloud service to give the data, why waste energy getting consent from its owner! More bugfinding AI, more end to end encryption, more CVEs and more fixes, cannot happen soon enough.

dangoodmanUT

> In April, Anthropic announced a new model called Mythos that was optimized for software vulnerability finding No, it was just good at it because it wasn't RL'd against it. I know this is a small detail, but it tosses journalistic credibility in my eyes.

hn_submit

I predicted a long time ago that if computers become unhackable LEA and intelligence agencies will push for laws that require backdoors to be built into hard- and software. It will be interesting to see if my prophecy becomes reality. BTW I also hate that Hacker News is being dominated by articles on A.I. lately. Maybe we should vote on HN reducing or even eliminating A.I. related news?

teravor

I don't think the thesis that a government will be able to do something will ultimately hold. I don't see how they can avoid "going dark" in a democracy. we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software. if they pass laws that mandate backdoor access and block software which doesn't conform more and more people will move to the dark networks. and if they effectively block the dark networks (in the limit they will have to block all encrypted communications) then we will be living in a tyranny. freedom is messy. accept that digital crime can only be solved when the criminal makes a tangible mistake. LLM's will be building profiles on criminals to help with identifying mistakes.