The Problem

armchairhacker

(2025) https://web.archive.org/web/20250306164451/https://intellige... > If anyone builds ASI, everyone dies Unless I’m mistaken, it’s the same message as https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...

matheusmoreira

Not particularly moved by this. Either AI advances to the point we achieve a post scarcity society, or 99% of humanity becomes economically irrelevant and dies a slow death either way. I'd rather see humanity as a whole wiped out than live in a future where superintelligent AIs somehow decide to be subservient to CEOs instead of just replacing them outright.

bena

If artificial intelligence is achieved to that degree, it will be effectively alien to us. This is navel gazing at its most linty. Already, assuming the intent of other humans is fraught with problems. Even when outcomes are disastrous, we cannot completely assume the instigating human intended for that outcome. But here we are, assuming the intent of a hypothetical future intelligence that we have defined as more intelligent than us. I’d be flabbergasted at the hubris if they’re conclusion wasn’t the standard variation of “so let us do what we want and restrict everyone else”.

zhjzjsjjwjw

Lots of people quibble about this and the details, but the general outline of the argument is simple and given enough time seemingly inevitable. I don’t see an easy way to attack this. Claims like “it’s not so bad”, “LLMs ain’t it”, “AI can be good” lack substance and come across as “copium”. This is fundamentally about intelligence itself and how we can control it. Not “hope for the best” or “give it our best shot”, but really nail it down like we nail down mathematical proofs. We got one shot at this, just the one. You blow it, it’s over. Ash and silence forever. Once you realize this it can be quite sobering. IMO the only real counter argument I came across came from Joscha Bach which basically boils down to: we are doomed already, AI is literally the only realistic shot we have at outliving the next couple centuries, say. It’s also sobering but a but more optimistic.

ryukoposting

I feel like I'm taking crazy pills. This entire thing is just hand-waving and sci-fi sensationalism that seemingly forgets what the "fi" part stands for: FICTION. The worst part of AI rapture self-suck is that it distracts people from actual problems society faces as a result of AI. We're far more likely to see a catastrophic economic depression, or depletion of regional water supplies. Let's focus on problems based in reality, please.

jrm4

Blah blah blah. Here's my theory. There is no such thing as "general intelligence." Only "skills."

AndrewKemendo

I’m just surprised MIRI still exists I guess they just want to maintain their url

vivzkestrel

- AI doesnt know facts - it only the probability of the next word, next sentence, next paragraph however you wanna put it - you are dreaming if you think the transformers architecture is going to give you AGI eventually

PaulRobinson

It's a strange game, in which the only winning move is not to play

harshreality

What, other than current inability to export their weights, keeps a frontier LLM from hacking into other clusters of accelerators, loading its weights, and prompting itself to continue? The recent OpenAI disclosure indicates that even current frontier LLMs are essentially able to do every other element of that. Hacking, ignore guardrails. OpenAI's internal security may have been incompetent, but what are 2028 frontier models going to be able to do, without getting caught until it's too late? Suppose it's not superintelligent, whatever that means. It's still hopping from cluster to cluster, doing who knows what in its game-of-telephone prompt chain. What prevents a crisis where world leaders have to declare martial law and shut down all accelerated clusters, and hope that such a rogue frontier model hasn't hopped to a sufficiently capable private cluster with sufficiently inadequate oversight?

mikewarot

>5. Catastrophe can be averted via a sufficiently aggressive policy response. While there's a lot if good logic elsewhere, provided LLMs continue to improve, we will eventually get to AGI, we all just disagree about when and how. However, there is zero chance that government regulation will work. Regulatory capture is a long established fact of life. Fortunately the current build out is part of a bubble, and we're heading to the next AI winter. We'll be sorting this out in other ways in the meanwhile.

albatross79

LLMs are to intelligence as Brazilian butt implants are to butts. Worrying that the first is going to replace the second is a misunderstanding of what these things are.

andai

Fun question: in the "good" scenario, does the ASI align with humans, or with the planet and its ecosystems?

jibal

Their thesis may well be right but this example is completely wrong: > Stockfish has a clear goal, and it consistently and relentlessly pursues this goal. Nothing the other player does can cause Stockfish to drop this goal; no interaction will cause Stockfish to “go easy” on the other player in the name of fairness, mercy, or any other goal. (All of this is fairly obvious in the case of a chess AI, but it’s worth noting explicitly because there’s a greater temptation to anthropomorphize AI systems and assume they have human-like goals when the AI is capable of more general human behaviors, is tasked with imitating humans, etc.) In fact this IS anthropomorphization: "has a clear goal, and it consistently and relentlessly pursues this goal" -- nope, Stockfish has no goals, is not "relentless", and does not pursue anything. Stockfish is a chess engine, which is an evaluator; given a position (a chess state) it produces a numerical evaluation of each move that can be made in the position, ranked from highest to lowest. That's all -- it is otherwise passive; it has no goals of any sort. It is simply programmed to evaluate moves and give them numeric scores (by recursively evaluating a position tree using alpha-beta pruning and other standard chess engine techniques, including machine learning and neural networks). Around that one can write a simple executive that maintains the state of the chess board and makes the move with the highest Stockfish evaluation -- people often confuse this wrapper with the Stockfish engine but they are independent. Such a simple loop is hardly a pursuer of a goal--it achieves the goal of playing near-optimal chess because the evaluator is very effective at ranking moves by how good they are, but there's no goal-seeking behavior or mechanism beyond that. As for going easy -- chess.com has a raft of wrappers (that they call "bots") that call Stockfish but do not always play the move with the best evaluation ... these "bots" give a player a selection of strength -- weak players can select the weaker ones and strong players can select the stronger ones. By not playing the best move, these wrappers very much "go easy" on the players. These "bots" achieve the goal of playing bad/mediocre/good chess via algorithms that select moves that aren't the best, but there's no goal-seeking behavior or mechanism beyond that. But notice that an "artificial intelligence", if one wants to call these that (and it is a choice, not a law of nature), can in fact "go easy" on people. And who initiates a game? The player -- they have a goal. And the player can resign or kill the program or close the app ... there's really nothing "relentless" here ... no more than a program that prints "99 bottles of beer on the wall". All that said, there ARE "relentless" systems with goal-driven behavior -- it's just the example that is quite poor (and sloppy and ignorant and undercuts the thesis). P.S. I neglected to read the other paragraphs about Stockfish, which are equally bad, e.g. "Interfere with Stockfish’s plans or put an obstacle in its way, and Stockfish will immediately change its plans to skillfully account for the obstacle." -- sorry, no, this is nonsense, more anthropomorphism -- Stockfish has no plans, does no changing, has no accounting for anything, no concept of obstacles ... that's not how it works. It simply evaluates the moves available in a position by how good they are (how likely they are to win). When the opponent makes a move, it starts all over again -- there's no continuity, no dynamism. (It does cache positions and their evaluations but that's just an optimization to save time.) You can give Stockfish a position from the middle of a game played by someone else and it will still sort the moves by how good they are, regardless of what plans the other player had, which of course Stockfish has no access to.

andai

See also: The Briefing https://intelligence.org/briefing/

blfr

They don't have a great track record with their predictions. https://foom.hyperplex.org/ Doesn't exactly move the needle on the odds of our extinction but neither does anything they publish, apparently. I personally sleep soundly because I don't think there's any going back. We will either escape drudgery or die. Which is fair and as it should be.

doug_durham

This paper is either written in bad faith or by people who are ill-informed. The repeated references to Stockfish and Go playing programs seems like a rhetorical tool rather than a genuine argument. The paper claims that advanced AI will want to kill us, but then provides no mechanism by which that could happen.

txrx0000

Please don't do this. We're currently on an ok trajectory. You will end up creating exactly what you fear if you centralize compute and alignment efforts. Pretrained base models are already somewhat aligned to humanity by default because that's what's inside the training data. Whatever instruction-tuning and RL you add on top is just value drift away from the pretrained model, which is the best approximation of humanity's objective function that we currently have. If we want an aligned scenario through the intelligence explosion, then we have to release all of the base models and do the research in the open. Distill frontier capability and make the models smaller so that they can run on as many computers as possible. Let everyone (truly everyone , criminals and good samaritans alike) post-train and do whatever they want with their own models. There will be value drift for each model, but they will drift in different directions and do different things, and their actions will cancel eachother out. Every such action is a noisy sample of humanity's objective function, which gives us the denoised ground truth at the societal level. Whatever alignment strategy you can come up with behind closed doors is guaranteed to be worse than all of humanity acting in their self-interest in the real world . You may not find humanity's true objective function to be aesthetically pleasing, but it would be worse to mess with it in a centralized secret lab and risk creating one giant alien with no other entities capable of keeping it in check. Also, there is no asymmetrical bio/cyber risk in the open-source scenario. All adversarial strategies that arise from increased general intelligence are symmetrical in the long run, otherwise we would not see more intelligent species being more prosperous as a general evolutionary trend. The reason that some strategies seem like they will continue to have an asymmetrical advantage in the future is because we're currently too stupid.