The argument is a fundamentally flawed way of using probabilities, a la Zeno’s paradox. One could also argue that we are doomed by AI because there are multiple ways AI could kill us, each with a probability, we are dead if only one of them is true, and there’s only an infinitesimal chance all the dice rolls come out in our favor.
The core argument for Doomerism doesn't rest on most of these assumptions at all. It's much more basic: if we create another intelligent species, that is significantly smarter than us, and reproduces faster than us what happens to us is no longer within our control. The doom here is not that humans get exterminated in some Terminator scenario, it's that we become pets/zoo objets/creatures confined to a nature preserve, while AI takes over the universe. And the only way that doesn't happen is if there is some fundamental magic preventing the AIs from being significantly more capable than humans. Everyone wants to analogize AI to technology, but that analogy *assumes away* exactly the thing that concerns people about AI.
Of these eight arguments, only 7 isn't one where the doomers won't immediately come after you with a mix of laugh and specific arguments why probabilities of all of these are infinitesimal. And their estimate for 7 is likely to be much lower, too (partly due to "accounting for unknown unknowns", i.e. discounting due to possible paradigm shifts).
(In particular, 3 is denying "instrumental convergence": nearly any goal will find useful more influence as a subgoal, and someone who thinks "women" - or, at least, stereotypical women - are an exception to evolving this subgoal hasn't much contacted with them in situations where they want something; 5 is "human vs. ant"; and so on and so forth.)
This is not to say that the argument is necessarily wrong, just that none of these have been missed by the doomers.
One potential issue with this style of analysis is that you could run it the other way and consider the various paths to some kind of AI doom/catastrophe. Then you might get a lot of small probabilities adding up to a large number.
I’m not sure how to resolve the tension between these two styles of reasoning.
> I think a 4% chance that AI wipes us out and we can do something about it is high enough that people should work on the problem, and I’m glad some do.
I think that's compatible with the AI doomerism view, so it doesn't seem like you actually disagree with their core point? Once there is agreement on this core point, we can then quibble on whether slowing down AI research is a viable solution etc., but I think opening your post with a claim that you disagree with AI Doomerism is misleading. It's not necessary for the probability of the extinction of the human race to be "high" (whether that means to you something like "more than 50%" or "more than 0.1%" or whatever) for the AI Doomerism argument to be compelling.
My issue with the main crux of the article is that all those probabilities are likely to be correlated, you can't just multiply them all by each other, that only works when the things in question are wholly independent. You could make a pro-doomer argument along similar lines by racking up pro-doomer arguments
A smaller correction
" it’s difficult to think of many technologies that people have voluntarily used that did not make the world better off in the long run"
This is simply survivorship bias. I'm sure there is a random technology invented in the 1800s that just didn't so everyone forgot about it. You also can't assume that the technology in question will be imposed only voluntarily. If someone creates an AI capable of self-improvement, and in fact does so while being misaligned, despite me never agreeing to be affected by it, its need for resources will affect me
I agree that anyone > 50% is probably thinking about this incorrectly.
But I disagree that 4% risk is not vitally important, because all of that risk is probably spread over the next 10 years. It's the single most important thing that will happen over that time.
I also think your entire system is backwards. It's not "these are reasons why something won't happen" because you can just keep thinking of more and more and more and more until you end up at whatever arbitrarily low number you desire. You have to do it the other way around - these are why something COULD happen. You'll quickly run out of reasons that aren't straightforwardly stupid.
(I happen to think that the risk is more like 30% over the next 10 years: we're about to make something much smarter than us, it will have different goals to ours, we don't know how to change those goals, and it might achieve them. I think all of those statements are true to various probabilities, and I think we need all of them to be true for there to be enough risk)
Also, one should consider that the possible “doom” scenarios are much more varied than the ones commonly predicted by prominent AI “doomers” (i.e., one or few uncontrollable AIs taking over). And many of those are quite compatible with one or more of the “optimistic” assumptions being true.
In particular, even with reasonable controllable AIs, it’s not at all clear what chance of survival most of humanity would have in a world where increasingly many decisions are delegated to inscrutably super-intelligent AIs and where humans are no longer able to earn above-subsistence wages. (Yes I’m aware of the comparative advantage argument, but that still depends on the marginal cost of human-equivalent compute and robots somehow remaining higher than that of human labor, an assumption that seems dubious at best.)
4% risk of dying (that I can prevent) would be the ONLY thing I cared about if it affected me as an individual. Would you eat food that had a 4% chance of being lethal poison? Would you play Russian roulette with a gun with 1 bullet out of 25? Even setting aside the rest of humanity, a 4% chance of dying is HUGE. And if all of humanity is dying alongside me, I care even more than I would if it were only me at risk.
Your 4% chance is still _really high_ when we're talking about existential risk. It easily justifies spending billions on AI Safety research and a regulatory apparatus that gives us a near-guarantee it is existentially safe, at the cost of slowing down capabilities research.
I wouldn't get on a plane with a 4% chance of a crash. When the odds of existential risk from AI disaster are closer to the odds of my next flight crashing to the ground, I'll be happy hop aboard the full steam ahead AI train.
I've placed the odds at 5-10% myself for the last couple of years or so, and I also sign on to your 1-5% "risky but we can do something about it" assessment. So I agree with your overall conclusion about what the risk is. I just think that's an _obvious_ sign we need to be more cautious.
There are two real arguments: (1) an out-of-control AI superintelligence won't happen, and (2) if it does happen, it won't decide to kill us. I really think each of those is worth evaluating on its own merits, rather than spamming arguments on other side.
My bet is mostly on (1) because I think (2) is effectively unknowable. And the most likely path for (1) is this: AI-related investments stop paying off, because each improvement requires an exponential increase in resources. At some point companies like MSFT realize that the next major upgrade of AI will be 5-10% better for practical business purposes (which will see diminishing returns long before abstract measures of intelligence), but it will cost $1 trillion to build, and the payoff isn't worth it. And so no one bothers to build it.
Compare to something like manned space travel: it turns out that going into Earth's orbit is practical, for artificial satellites, space tourism, etc. After that, the productivity benefits of space travel drop off exponentially. We could send a manned mission to Mars, but it would cost more money than anyone wants to pay. So it never happens. If it ever does happen, it will be so incredibly costly and pointless that, just as we never went back to the moon, no one will want to try anything like it again for 50+ years.
Another worry is that as a non-technical person (as far as I know), it might be hard for you to assign a low probability on any of those individual statements which will inevitably make the total calculated probability of us being safe high. While it is much more likely that a technical person could assign a near zero probability to at least some of those statements.
The one thing I would take issue with is the idea that the smarter a person is the more moral. I bet Stalin had a pretty high IQ, as he read literature. Higher than the vest majority of peasants. But also very evil. This logic also applies to the Nazis and most dictatorships including the Islamic ones. So a super smart AI could not be counted on to be moral. It could, like Stalin, reason that certain classes or person were just too much in the way of a utopia.
Some people have already come up with (relatively) optimistic estimates of AI existential risk with a similar method of multiplying probabilities of several necessary conditions.
Notably, there was a study commissioned by an EA organization, which also came up with a single-digit percentage probability (although the author notes that he has updated towards a higher estimate in the meantime):
Yudkowsky and other pessimists have criticized this kind of reasoning as “multiple stage fallacy”, where you end up multiplying probabilities of each stage as independent events, whereas in reality it would require an extremely difficult estimate of conditional probabilities.
I think Yudkowsky originally coined this term in a different context years ago, in response to a 2015 article by Nate Silver that used similar reasoning to estimate an extremely low (2%) probability for Trump to win the Republican nomination:
In retrospect it seems clear that Silver’s reasoning was fallacious exactly along these lines, leading to a severe underestimate of the final probability. (And that’s from someone whose grasp of probability is generally first-rate!) This, I think, should make one very skeptical about analogous optimistic estimates of AI risk.
The argument is a fundamentally flawed way of using probabilities, a la Zeno’s paradox. One could also argue that we are doomed by AI because there are multiple ways AI could kill us, each with a probability, we are dead if only one of them is true, and there’s only an infinitesimal chance all the dice rolls come out in our favor.
The future will be science-fictional either way, we're just arguing over the subgenre.
The core argument for Doomerism doesn't rest on most of these assumptions at all. It's much more basic: if we create another intelligent species, that is significantly smarter than us, and reproduces faster than us what happens to us is no longer within our control. The doom here is not that humans get exterminated in some Terminator scenario, it's that we become pets/zoo objets/creatures confined to a nature preserve, while AI takes over the universe. And the only way that doesn't happen is if there is some fundamental magic preventing the AIs from being significantly more capable than humans. Everyone wants to analogize AI to technology, but that analogy *assumes away* exactly the thing that concerns people about AI.
Of these eight arguments, only 7 isn't one where the doomers won't immediately come after you with a mix of laugh and specific arguments why probabilities of all of these are infinitesimal. And their estimate for 7 is likely to be much lower, too (partly due to "accounting for unknown unknowns", i.e. discounting due to possible paradigm shifts).
(In particular, 3 is denying "instrumental convergence": nearly any goal will find useful more influence as a subgoal, and someone who thinks "women" - or, at least, stereotypical women - are an exception to evolving this subgoal hasn't much contacted with them in situations where they want something; 5 is "human vs. ant"; and so on and so forth.)
This is not to say that the argument is necessarily wrong, just that none of these have been missed by the doomers.
One potential issue with this style of analysis is that you could run it the other way and consider the various paths to some kind of AI doom/catastrophe. Then you might get a lot of small probabilities adding up to a large number.
I’m not sure how to resolve the tension between these two styles of reasoning.
You conclude:
> I think a 4% chance that AI wipes us out and we can do something about it is high enough that people should work on the problem, and I’m glad some do.
I think that's compatible with the AI doomerism view, so it doesn't seem like you actually disagree with their core point? Once there is agreement on this core point, we can then quibble on whether slowing down AI research is a viable solution etc., but I think opening your post with a claim that you disagree with AI Doomerism is misleading. It's not necessary for the probability of the extinction of the human race to be "high" (whether that means to you something like "more than 50%" or "more than 0.1%" or whatever) for the AI Doomerism argument to be compelling.
My issue with the main crux of the article is that all those probabilities are likely to be correlated, you can't just multiply them all by each other, that only works when the things in question are wholly independent. You could make a pro-doomer argument along similar lines by racking up pro-doomer arguments
A smaller correction
" it’s difficult to think of many technologies that people have voluntarily used that did not make the world better off in the long run"
This is simply survivorship bias. I'm sure there is a random technology invented in the 1800s that just didn't so everyone forgot about it. You also can't assume that the technology in question will be imposed only voluntarily. If someone creates an AI capable of self-improvement, and in fact does so while being misaligned, despite me never agreeing to be affected by it, its need for resources will affect me
I agree that anyone > 50% is probably thinking about this incorrectly.
But I disagree that 4% risk is not vitally important, because all of that risk is probably spread over the next 10 years. It's the single most important thing that will happen over that time.
I also think your entire system is backwards. It's not "these are reasons why something won't happen" because you can just keep thinking of more and more and more and more until you end up at whatever arbitrarily low number you desire. You have to do it the other way around - these are why something COULD happen. You'll quickly run out of reasons that aren't straightforwardly stupid.
(I happen to think that the risk is more like 30% over the next 10 years: we're about to make something much smarter than us, it will have different goals to ours, we don't know how to change those goals, and it might achieve them. I think all of those statements are true to various probabilities, and I think we need all of them to be true for there to be enough risk)
Also, one should consider that the possible “doom” scenarios are much more varied than the ones commonly predicted by prominent AI “doomers” (i.e., one or few uncontrollable AIs taking over). And many of those are quite compatible with one or more of the “optimistic” assumptions being true.
In particular, even with reasonable controllable AIs, it’s not at all clear what chance of survival most of humanity would have in a world where increasingly many decisions are delegated to inscrutably super-intelligent AIs and where humans are no longer able to earn above-subsistence wages. (Yes I’m aware of the comparative advantage argument, but that still depends on the marginal cost of human-equivalent compute and robots somehow remaining higher than that of human labor, an assumption that seems dubious at best.)
4% risk of dying (that I can prevent) would be the ONLY thing I cared about if it affected me as an individual. Would you eat food that had a 4% chance of being lethal poison? Would you play Russian roulette with a gun with 1 bullet out of 25? Even setting aside the rest of humanity, a 4% chance of dying is HUGE. And if all of humanity is dying alongside me, I care even more than I would if it were only me at risk.
Your 4% chance is still _really high_ when we're talking about existential risk. It easily justifies spending billions on AI Safety research and a regulatory apparatus that gives us a near-guarantee it is existentially safe, at the cost of slowing down capabilities research.
I wouldn't get on a plane with a 4% chance of a crash. When the odds of existential risk from AI disaster are closer to the odds of my next flight crashing to the ground, I'll be happy hop aboard the full steam ahead AI train.
I've placed the odds at 5-10% myself for the last couple of years or so, and I also sign on to your 1-5% "risky but we can do something about it" assessment. So I agree with your overall conclusion about what the risk is. I just think that's an _obvious_ sign we need to be more cautious.
Here's a more rigorous version of this: it's not just about having longer series of probabilities, but trying to get them to be independence. https://www.strangeloopcanon.com/p/agi-strange-equation
There are two real arguments: (1) an out-of-control AI superintelligence won't happen, and (2) if it does happen, it won't decide to kill us. I really think each of those is worth evaluating on its own merits, rather than spamming arguments on other side.
My bet is mostly on (1) because I think (2) is effectively unknowable. And the most likely path for (1) is this: AI-related investments stop paying off, because each improvement requires an exponential increase in resources. At some point companies like MSFT realize that the next major upgrade of AI will be 5-10% better for practical business purposes (which will see diminishing returns long before abstract measures of intelligence), but it will cost $1 trillion to build, and the payoff isn't worth it. And so no one bothers to build it.
Compare to something like manned space travel: it turns out that going into Earth's orbit is practical, for artificial satellites, space tourism, etc. After that, the productivity benefits of space travel drop off exponentially. We could send a manned mission to Mars, but it would cost more money than anyone wants to pay. So it never happens. If it ever does happen, it will be so incredibly costly and pointless that, just as we never went back to the moon, no one will want to try anything like it again for 50+ years.
Another worry is that as a non-technical person (as far as I know), it might be hard for you to assign a low probability on any of those individual statements which will inevitably make the total calculated probability of us being safe high. While it is much more likely that a technical person could assign a near zero probability to at least some of those statements.
The one thing I would take issue with is the idea that the smarter a person is the more moral. I bet Stalin had a pretty high IQ, as he read literature. Higher than the vest majority of peasants. But also very evil. This logic also applies to the Nazis and most dictatorships including the Islamic ones. So a super smart AI could not be counted on to be moral. It could, like Stalin, reason that certain classes or person were just too much in the way of a utopia.
Some people have already come up with (relatively) optimistic estimates of AI existential risk with a similar method of multiplying probabilities of several necessary conditions.
Notably, there was a study commissioned by an EA organization, which also came up with a single-digit percentage probability (although the author notes that he has updated towards a higher estimate in the meantime):
https://arxiv.org/abs/2206.13353
Yudkowsky and other pessimists have criticized this kind of reasoning as “multiple stage fallacy”, where you end up multiplying probabilities of each stage as independent events, whereas in reality it would require an extremely difficult estimate of conditional probabilities.
I think Yudkowsky originally coined this term in a different context years ago, in response to a 2015 article by Nate Silver that used similar reasoning to estimate an extremely low (2%) probability for Trump to win the Republican nomination:
https://forum.effectivealtruism.org/posts/GgPrbxdWhyaDjks2m/the-multiple-stage-fallacy
In retrospect it seems clear that Silver’s reasoning was fallacious exactly along these lines, leading to a severe underestimate of the final probability. (And that’s from someone whose grasp of probability is generally first-rate!) This, I think, should make one very skeptical about analogous optimistic estimates of AI risk.