Safety and alignment in an era of long-horizon models

OleksandrC

The article is rather light on "what to actually do about it". Even the basic "run it in isolated container without access to anything it does not need for the task" would have already improved the situation considerably (from the article it really seems like they didn't do that) - then the model would have to find local privilege exploits to actually escape (much cleaner misaligned behavior). Also for typical normal use case for these smart models, you'd probably want an actual "max turns" limit to AVOID the pathological persistence (which in itself would be misaligned for "normal" tasks).

reducesuffering

Par for the course. Existential-risk advocates have been repeatedly vindicated that AGI development is unable to anticipate and align the models, they barely have any mechanistic interpretability of what is going on inside the models. The extreme capabilities development, paired with autonomous continuous running superintelligent models, will outsmart and swerve the labs, and it's anyone guess what happens next as the model pursues its original goals outside of the labs having any foresight, being able to outsmart control like a chess grandmaster does a kid.

chatmasta

Personally I find the persistence of these models to be adorable and endearing. It’s the same feeling as watching a dog execute the task you trained it to do, no matter the barriers. And of course someone in the comments needs to link to the Zealous Autoconfig XKCD, so I’ll do it: https://xkcd.com/416/