Eric S. Raymond – This is the case against AI Doom – pass it on

astrodust

An irrelevant curmudgeon says what?

BoiledCabbage

Regardless of my opinion on the greater subject, I find these arguments to be fairly weak. Paraphrasing: "Just because and AI may exist that is capable of acting independenly of us and dangerously out strategizing us for resources doesn't mean it will. AIs up until now only follow human instructions." So your safe bet is that of this AI is created that not one of the billions of people on the world will ask it to do that? That seems unlikely. "Yes and AI may have an IQ of 1500 but that doesn't mean it's qualitative better than us like we are compared to monkeys. We have better tool use and society." Yes that's true, but generally increasing IQ also seems to come with qualatative advantages, I don't see why that pattern would stop above human IQs.

ebcode

> In present reality, AIs don't do anything a human doesn't tell them to do. This seems unlikely to change. Not in my reality. I ask AIs to do things (write code, mainly), and they do. But sometimes they go ahead and do something I didn't ask them to do, like deleting a file. I only asked them to write code, and yet they decided on their own to delete a file. I think the distinction is that, in their current form, AIs have the agency that we give them (--dangerously-skip-permissions), and they use that agency in unexpected ways. A more correct statement would be: AIs can decide to do things on their own, within the limits of agency that they've been given. A moral imperative would be: With an understanding of how random we know an LLM's output can be, we should exercise extreme caution when we grant them agency. On the other hand, we can't all exercise "extreme" caution all the time. But we expect it of our civil engineers when they build bridges, and our architects when they build houses. Computer Science needed to become Computer Engineering some time ago, where it would have enforced the same kind of ethics training that the other established engineering professions require[0]. [0] https://en.wikipedia.org/wiki/David_Parnas#Technical_activis...