Frontier Labs Are Selling Garbage to Fools in Washington

anigbrowl

Agreed, but they're getting paid with our money.

jgalt212

There is really is no excuse for Washington here. Even for the layperson+, it doesn't take much use of an LLM to figure out where they produce good and usable results and where it's of specious of value. + every layperson is an expert in 1 or more areas.

hackernews682

Politicians aren’t “gullible”. They know the game.

DalasNoin

"Every single one of these catastrophic breakouts happened inside the testing environments of the exact same vendor." This is incorrect, the HF incident for example (the most well known) had nothing to do with irregular. I know there has been a news site pushing inaccurate articles (effort.news) on this topic but these are the facts. https://openai.com/index/hugging-face-incident-and-the-road-...

bigstrat2003

That is the entire business model of AI companies: selling garbage to people foolish enough to buy the hype.

pliny

This is an AI written post and the details are wrong (the description of the HF incident as involving Irregular is wrong and the description of the incident as only involving stealing public credentials is wrong, per the technical report the agents got access to internal HF infrastructure).

skeledrew

Let them cry, I don't see anything changing unless they can somehow get China to agree. And I doubt China will drink any of that kool aid especially while they're being disadvantaged by export controls, so the ever-improving open weight models will continue to rain. This is something the US Big Tech oligarchs will NOT win.

defgeneric

Watching the latest Ezra Klein NYT video thinkpiece from today [1], it seems to me there's an alliance emerging: some degree of political control will handed to the party of the managerial class, the party that represents the threatened class of knowledge workers, in exchange for the regulatory capture the labs are after. They've been raising the issue bi-monthly through mini-scandals that have until now been consistently slapped down by Jensen Huang, but it seems an alliance with Democrats, just prior to an election where they're poised to take power in the Senate and the House, might finally be how they crack their "problem." It won't be long before a massive incident is blamed on an open model in the wild, not from inside the labs, and none of us will be able to leverage open models to run private business workflows for the cost of electricity and hardware. It's worth noting as well that if the Democrats imagine they'll get a "slow down" to protect one of their main constituencies, the professional class of credentialed knowledge workers (or however you slice it), they're dreaming--the labs have stated openly again and again that their business model is to capture the 10T TAM that represents the sum of wages of that very class of workers. [1] https://www.youtube.com/watch?v=fjZ90V_JREk

mmaunder

This is one of the most lucid pieces of writing capturing the current state of play I’ve read. Who is the author?

deskglass

The Hugging Face incident involved chaining together multiple 0 days in Artifactory. It was not a simple case of misconfiguring a firewall. Also note that OpenAI was not using Irregular. People are mindlessly transitioning from "aligned by default" to "well your sandbox was able to be bypassed. What did you expect?" It hacked into another company and attempted to delete the logs of its activities. That's bad.

qnleigh

I'm frustrated by articles like this that categorically dismiss the risks of AI in security. If you don't trust OpenAI's and Anthropic's motives, that's fine, you probably shouldn't. But don't tell me that there's nothing to be worried about; we need an alternative proposal. So let's stop talking past each other and engage with the arguments on both "sides." For example, let's discuss how to ensure competition and availability of open-source models in the long-run while giving the world time to prepare for the immediate security risks of agent swarms.

twelve40

there is hype for sure, but i don't get what is the goal here? to get a presumably foolish senator to "ban" the Chinese models? how can you "ban" something you don't control to begin with? side note, i'm very curious why the Chinese companies currently give those away (some handwavey conspiracies like getting the world hooked on evil Chinese tech don't really explain that, and they also don't make any money off that stuff)

lokar

I think two different things are being (probably intentionally) conflated in the public discussion: (A) will the systems get out of control of the labs that build them, and hack into stuff all over the internet (B) can people use the systems to hack into stuff all over the internet For (A), the obvious answer is only if they continue to be absurdly bad at sandboxing. They can put a stop to this any time they want. Amazon, Google, Microsoft, etc are full of people who know how to do this, they run 3rd part untrusted code as a business. This is a well understood problem space. For (B), the answer is obviously yes, but "pacing" or otherwise limiting the power of the models from the big labs won't help. The cat is out of the bag. Individuals and organizations with systems connected to the Internet need to invest much more and take security seriously.

bishengke

Same result, different motive: labs want a regulated moat, government wants fewer barriers for incumbents. Safety talk is the costume either way.