AI Doomerism as Science Fiction

Judge anti-doomers by the totality of their arguments, not the average quality of each one

Intellectually, I’ve always found the arguments of AI doomers somewhat compelling. Yet instinctually I’ve always thought they were wrong. This could be motivated reasoning, as I find the thought of having to quit talking about what I’m interested in and focusing on this narrow technical issue extremely unappealing. But for years I have just had a nagging suspicion that from an objective perspective doomers are not forecasting the future correctly.

After much thought, I think I’ve finally understood why I don’t believe those who think that AI has a good chance of killing us all because it will have a utility function that is misaligned with human values. My reasoning is simple, but I’ve never seen it put this way before. For AI doomers to be wrong, skeptics do not need to be correct about any particular argument. They only need to be correct about one of them, and the whole thing falls apart.

Let’s say that there are 10 arguments against doomerism, and each only has a 20% chance of being true. That means that the probability that doomers are wrong is 1 - (1-0.2)^10 = 0.89. Each individual argument against doomerism might be weak, but the overall case against it is strong. If you play with the percentages and number of arguments, the probability of doomers being correct varies as in the graph below.

I don’t think we’re good at judging the inherent plausibility of individual arguments related to the alignment question, as smart people disagree strongly on nearly all of them. Here are eight views that don’t sound crazy to me, with attached probabilities that I just made up.

1. There are massive diminishing returns to intelligence, so a super AI being smarter than us is not going to have that much of an advantage. (25%)

2. Alignment will turn out to be a relatively trivial problem. Like just tell the LLM to use some common sense and it won’t turn us into a ham sandwich when we ask it to “make me a ham sandwich.” (30%)

3. You’d need to program in power-seeking behavior in order to get it. Pinker: “There is no law of complex systems that says that intelligent agents must turn into ruthless conquistadors. Indeed, we know of one highly advanced form of intelligence that evolved without this defect. They’re called women.” (25%)

4. AI will simply be benevolent. Perhaps there is some law in which there is an intimate connection between reason and morality. This is why humans are more moral than other animals, and the smartest humans and societies are more ethical than the dumber ones. (10%)

5. Super AI will find it useful to negotiate with us and keep us around, just as how humans could form mutually beneficial relationships with ants if we could get over the communication barrier. Katje Grace has made this argument. (10%)

6. Super AIs will be able to check and balance one another, which can create dangers for humanity, but opportunities for it to thrive. (10%)

7. Research into artificial intelligence will stagnate indefinitely, perhaps because it runs out of training data, or some other reason. (35%)

8. We’re fundamentally mistaken about the nature of this thing “intelligence.” There are domain specific abilities, and we’ve been misled by metaphors derived from the psychometric concept of g. See also Hanson on “betterness.” (35%)

One issue here is you can divide the arguments however you want. So “alignment is a relatively trivial problem” can be broken down into all the ways that we might solve alignment. Moreover, isn’t 3 just a subset of 2? And 1 and 8 are conceptually pretty close. But I think that there are more ways to divide arguments and make more of them than there are to combine them. If I actually spelled out the ways alignment could be solved and connected a probability to each one, the case for doomerism would look worse.

There’s no perfect way to conduct this analysis. But based on my arbitrary probabilities given to the eight arguments above, we can say that there is an 88% chance doomers are wrong. A 12% chance humanity ends still seems pretty high, but I’m sure there are many other arguments that I can’t think of right now that would bring that number down.

There is also the possibility that although AI will end humanity, there isn’t anything we can do about it. I would put that at maybe 40%. Also, one could argue that even if a theoretical solution exists, our politics won’t allow us to reach it. Again, let’s say that is 40% likely to be true. So we are down to a 12% chance that AI is an existential risk, and then a 0.12 * 0.6 * 0.6 = 4% chance AI is an existential risk and we can do something about it.

I’ve always had a similar problem with the Fermi paradox, and also the supposed solution to it that says we know that UFOs have actually visited earth. On the latter claim, to think we’ve seen or made contact with aliens, you have to believe at the very least that:

1. Super intelligent aliens exist.

2. They’d want to visit us.

3. Them visiting us from beyond the observable universe would be consistent with the laws of physics.

4. Visiting us would be cost-effective based on their priorities and economic constraints

5. Despite them being willing and capable enough to make it here, they’re so incompetent that they keep getting caught, but not in a way that makes it too obvious.

Number 5 is the one that I have always found funniest of all. I can imagine aliens that can travel across the stars, but not ones that can do that but keep leaving evidence behind in the forms of things like grainy photographs, and nothing that would provide more direct evidence that doesn’t rely on firsthand testimonials. By a remarkable coincidence, our initial interactions with alien life provide a level of proof that could make its way into the National Enquirer but not meet the standards of real journalism. A few years ago there was UFO-mania across the media, and there’s been much less coverage of such reports recently being debunked by the Pentagon.

Yes, I know that the Fermi paradox doesn’t predict that any particular civilization will make itself known to us, but says that given how vast the universe is, at least one should be willing and able to. Yet positing infinite civilizations to me doesn’t solve the problem, which is that any aliens that exist must be very far away, and there are also infinite ways things can go wrong before they or their messages reach us.

Now, I think a 4% chance that AI wipes us out and we can do something about it is high enough that people should work on the problem, and I’m glad some do. But I take issue with those who seem to believe that the existential risk is more likely than not, or even that it’s obviously true that this will happen if we follow a few maxims like “intelligence helps you accomplish things” and “intelligent beings seek control over resources.”

We must also consider that there is potentially a cost to freaking people out. My view is that if AI doesn’t destroy us, it will be a massive boon for humanity, based on the fact that it’s difficult to think of many technologies that people have voluntarily used that did not make the world better off in the long run. Thus, AI doomerism isn’t something we can indulge in without worrying about the downside. If this crowd scares enough influential politicians and business leaders, there’s a possibility that they delay a better future indefinitely, with nuclear power being an important precedent here. Not maximizing intelligence could itself pose an existential risk to humanity, as we just continue being stupid until civilization is wiped out by a superbug or nuclear war. As long as AI doomers get jobs as programmers and look for narrow technical fixes instead of lobbying for government control over the industry, I applaud their efforts.

It’s a mistake to judge the case against AI doomerism by the quality of its average argument. The side that has an elaborate story to tell has the burden of proof to show that its imagined scenario is plausible. We can dismiss most science fiction as unable to provide good forecasts of the future even if we can’t say exactly how each piece of work is wrong or what paths future political and technological developments will take. A similar skepticism is justified for all stories involving alien intelligences, whether machines or lifeforms from other planets.