AI doomerism has been gaining currency after the Hugging Face incident. To get caught up, and find takes on why this is scaring people, see Ajeya Cotra, Dwarkesh, Scott Alexander, Zvi, and Ajeya talking to Dwarkesh. I agree that there are reasons for concern. But, upon doing a deep dive into what happened and questions of AI safety more generally, I think the case for optimism is being undersold.
Imagine that when we tested cars for safety, we got rid of the seatbelts, airbags, sensors, automatic braking, and stability control. We then find that cars kill a lot of crash test dummies under these scenarios, and the lesson that people take away from this is that cars are inherently unsafe products.
This is pretty analogous to what’s happening with most of the more concerning AI security incidents. This doesn’t mean that we shouldn’t be worried at all, but it does suggest that some of the coverage may be too alarmist.
A second consideration that makes me think we’re being too pessimistic is that groundbreaking technologies, setting aside medical and health discoveries, practically always come with risk, and those risks usually manifest themselves quickly. As soon as cars, airplanes and railroads were invented and deployed, to say nothing of nuclear weapons and guns, people died.
In laboratory conditions, we now have examples of AIs lying, hacking, and cheating. Yet through all this, the number of people actively harmed by misaligned AI agents in the real world is approximately zero, and so is the number of serious attempts, though it is true that humans have deliberately used them for malicious ends. One possible counterexample I’ve found is the MJ Rathbun/OpenClaw incident, where a guy created an autonomous coding agent with a combative personality and very little human supervision that published a personalized attack on a person who rejected its proposed code changes. Given the way the agent was configured, it is hardly a clean example of a normally deployed AI spontaneously becoming misaligned.
Yes, there have been a few suicides, but these are generally freak incidents that are difficult to link directly to chatbots. The harms of AI aren’t even close to what you see with social media. So far as we know, AIs have never autonomously stolen money or financial assets. Maybe they’re not smart or persistent enough to do this yet. But there are a lot of AIs out there being deployed, and you would think that by now some system would try to engage in real-life aggression on its own, beyond hacking incidents that have caused little actual harm.
This is not a paywall. The rest of this article is available to free subscribers of the newsletter. Just leave your email below to continue reading.
This post is sponsored by Mechanize. AI alignment is going fine for now, but they’re trying to make sure that we stay on the right path. If you’re interested in being part of that effort, Mechanize is hiring software engineers at a base salary of $300K-$400K/year, plus substantial bonuses. They also have openings for other positions, including Recruiter ($200K). Here is their website, and you can apply here.
If we set aside the abstract issues about alignment, we're still putting a lot of trust in what you refer to as internal and external security layers.
We're seeing open-source models progressing very quickly. Usually a few months behind top-level models like OpenAI.
So even if we think alignment is completely possible in theory. There's nothing stopping evil actors such as Hamas, North Korea or just some crazy guy using an open-source model without any safety layers for a specifically malicious task.
To me "solving alignment" is creating a model smart enough to cause significant damage but with deep-coded safety features that can't be removed by anyone.
Otherwise we're just creating pocket nukes and sending them out into the world.