AI doomerism has been gaining currency after the Hugging Face incident. To get caught up, and find takes on why this is scaring people, see Ajeya Cotra, Dwarkesh, Scott Alexander, Zvi, and Ajeya talking to Dwarkesh. I agree that there are reasons for concern. But, upon doing a deep dive into what happened and questions of AI safety more generally, I think the case for optimism is being undersold.
Imagine that when we tested cars for safety, we got rid of the seatbelts, airbags, sensors, automatic braking, and stability control. We then find that cars kill a lot of crash test dummies under these scenarios, and the lesson that people take away from this is that cars are inherently unsafe products.
This is pretty analogous to what’s happening with most of the more concerning AI security incidents. This doesn’t mean that we shouldn’t be worried at all, but it does suggest that some of the coverage may be too alarmist.
A second consideration that makes me think we’re being too pessimistic is that groundbreaking technologies, setting aside medical and health discoveries, practically always come with risk, and those risks usually manifest themselves quickly. As soon as cars, airplanes and railroads were invented and deployed, to say nothing of nuclear weapons and guns, people died.
In laboratory conditions, we now have examples of AIs lying, hacking, and cheating. Yet through all this, the number of people actively harmed by misaligned AI agents in the real world is approximately zero, and so is the number of serious attempts, though it is true that humans have deliberately used them for malicious ends. One possible counterexample I’ve found is the MJ Rathbun/OpenClaw incident, where a guy created an autonomous coding agent with a combative personality and very little human supervision that published a personalized attack on a person who rejected its proposed code changes. Given the way the agent was configured, it is hardly a clean example of a normally deployed AI spontaneously becoming misaligned.
Yes, there have been a few suicides, but these are generally freak incidents that are difficult to link directly to chatbots. The harms of AI aren’t even close to what you see with social media. So far as we know, AIs have never autonomously stolen money or financial assets. Maybe they’re not smart or persistent enough to do this yet. But there are a lot of AIs out there being deployed, and you would think that by now some system would try to engage in real-life aggression on its own, beyond hacking incidents that have caused little actual harm.
This is not a paywall. The rest of this article is available to free subscribers of the newsletter. Just leave your email below to continue reading.
This post is sponsored by Mechanize. AI alignment is going fine for now, but they’re trying to make sure that we stay on the right path. If you’re interested in being part of that effort, Mechanize is hiring software engineers at a base salary of $300K-$400K/year, plus substantial bonuses. They also have openings for other positions, including Recruiter ($200K). Here is their website, and you can apply here.
Consider the fact that this incident essentially validated doomer predictions around e.g. automated hacking, swarm intelligence, cheating to maximize reward functions, and incentives AI systems have to engage in deceptive behavior. All that stuff which got dismissed as "just sci-fi" actually happened.
What other "sci-fi" doomer predictions might get validated in the future?
For example, how about the "first critical try" line of reasoning presented in the following youtube video?
https://www.youtube.com/watch?v=7wy3xyoXYt8
For sufficiently advanced AI systems, if our alignment work isn't up to snuff, activating the system could mean game-over for humanity. In that case, the sort of trial-and-error approach we use for most technologies won't work. Furthermore, alignment of superintelligent systems could represent a qualitatively different and substantially harder problem, with limited transfer of alignment techniques and learnings.
If this is true, let's think about your question about what it means to "solve alignment". "Solving alignment" would mean an approach that we have strong reason to believe would work, in principle, with high confidence/very high reliability.
From that perspective, "a situation where it would be very difficult to prove to the relevant regulators that AI won’t misbehave when it’s gone out into the world" sounds perfect! If the entire human species is at stake, let's at *least* apply the level of regulatory scrutiny that's applied to e.g. bridges and aircraft, which only kill a tiny subset of humanity if they fail.
IIRC, the original rationale for creating these AI chatbots was actually to provide a sort of "model organism" for AI alignment work. The creators of the chatbots argued that it would be very difficult to do AI alignment work in the abstract, without concrete AI systems to validate ideas and gather data. At this point, we've utterly surpassed that objective. Now we can sit down with everything we've learned and do the real work (while freezing the advancement of AI systems and enjoying their current benefits).
“Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer” - Jakub Pachocki, Chief scientist at OpenAI