The argument is a fundamentally flawed way of using probabilities, a la Zeno’s paradox. One could also argue that we are doomed by AI because there are multiple ways AI could kill us, each with a probability, we are dead if only one of them is true, and there’s only an infinitesimal chance all the dice rolls come out in our favor.
That argument has to grapple with the fact that there have been a lot of technologies and none have ever killed us all or even really come close in recorded history. In fact, they’ve basically all been good. If you have two arguments where both seem equally plausible to you and one would have been correct at every point in human history and the other would have been wrong, which do you go with?
that's just survivorship bias. If we invented a technology capable of wiping us out, then we wouldn't be around to discuss that technology has in the past wiped us out
This seems to be besides the point? Your argument can be made for anything. Therefore it gives no information. You can introduce a new argument about the burden of proof or something, but the original post doesn't make any sense.
You're right, of course, but only due to #7 (which I'd put at 98.5%). The other arguments are flawed and Yudkowsky has debunked all of them at least once, as he has also this last argument.
No, creating something smarter than us, even via technology, isn't at all like other technology we've created. Our common ancestor with chimps created termite fishing and human intelligence both. One of these is not like the other.
You can’t just “turn it off”. You’re assuming it will have a green LED that turns red if it decides to be evil. Humanity won’t know anything is amiss until the AI has already wiped us out.
I think the argument you're responsing to is incorrect, but for a different reason than your response. Your intitial reasoning suggests a low p(doom). The responding comment suggests a high p(doom). You can't simply resort to the null hypothesis - that ignores the fact that ostensibly equivalent reasoning is yielding opposite conclusions. It isn't a matter of two different arguments that sound equally compelling, but ostensibly, of the same reasoning yielding contradictory results - a paradox that must be resolved.
The resolution, I think, is that in your argument each probability can be arbitrarily low. In contrast, the probabilities of the different ways AI doom can occur are bounded, since they have to sum to less than 1, so they can't be arbitrarily high.
To restate that, in your initial argument, for AI doom to occur, a series of independent possibilities all must be incorrect. Each additional reason drastically lowers p(doom). There's no constraint stopping p(doom) from approaching 0; as the number of arguments accrue, the probability naturally approaches 0.
However, in looking at the probabilities of the various ways AI doom can occur, each probability must be tiny, since they're mutually exclusive. And since they therefore aren't independent, there's no reason to think that additional possibilities would lead to convergence on 1.
[That's why the comparison to Zeno's paradox is incorrect. In Zeno's paradox, an object asymptotically approaches a destination, but in adding all the possible variations of AI doom, we don't asymptotically approach 1 - we just asymptotically approach p(doom) - but that doesn't doesn't tell us anything about what p(doom) is.]
Suppose astronomers spotted a 100-mile long alien starship approaching Earth. Surely this counts as a radically uncertain situation if anything does; we have absolutely no idea what could happen. Therefore - the alien starship definitely won’t kill us and it’s not worth worrying? Seems wrong.
What’s the base rate for alien starships approaching Earth killing humanity? We don’t have a base rate, because we’ve never been in this situation before. What is the base rate for developing above-human-level AI killing humanity? We don’t . . . you get the picture.
You can try to fish for something sort of like a base rate: “There have been a hundred major inventions since agriculture, and none of them killed humanity, so the base rate for major inventions killing everyone is about 0%”.
But I can counterargue: “There have been about a dozen times a sapient species has created a more intelligent successor species: australopithecus → homo habilis, homo habilis → homo erectus, etc - and in each case, the successor species has wiped out its predecessor. So the base rate for more intelligent successor species killing everyone is about 100%”.
The Less Wrongers call this game “reference class tennis”, and insist that the only winning move is not to play. Thinking about this question in terms of base rates is just as hard as thinking of it any other way, and would require arguments for why one base rate is better than another.
That was a very intelligent comment and a pleasure to read.
Also I hadn't encountered the base rate argument in terms of wiping predecessor species. That seems worth looking into in greater detail -- why exactly do the ancestors disappear when the successor species become dominant?
> However, in looking at the probabilities of the various ways AI doom can occur, each probability must be tiny, since they're mutually exclusive. And since they therefore aren't independent, there's no reason to think that additional possibilities would lead to convergence on 1.
Suppose there is a 50% chance of AI killing us in 2040, and if it doesn't, there is a 50% chance of it killing us in 2041 and so on for all years into the future.
That looks like converging to 1 to me.
I long list of "If the previous problem hasn't already killed you, this one is likely to" can converge to 1.
A series of fractions being infinitely long doesn't necessitate it converging on 1. For example, the series 1/2 + 1/8 + 1/32 ... converges on 2/3.
The same is true for probabilities. 1 is the most probabilites of an event can sum to, but that doesn't mean that all events have a probability of 1, even if there are infinitely many ways they can occur.
We can illustrate this with probabilities by considering an event we consider very unlikely, e.g. a random brick killing you. There are probably an infinite number of ways that could occur, but that doesn't mean that the probability is 1.
Richard's point is that arguements that preclude AI doom are largely independent, so each needs to be incorrect for doom to be possible (not just for doom to occur, incidentally; that is, they form an upper bound for p(doom), not an estimate for it), so as the number of arguments accrue, p(doom) appoaches 0.
This isn't just a semantic game that can be applies to any values or any probabilities. As the series demonstrated, not all series converge on 1. And as the brick example illustrated, an event, however unlikely, can have its probability space partitioned infinitely, but that doesn't make the event more likely, and it certainly doesn't give it a probability of 1.
One could give an example of a series of probabilities that converges to 1, and then label the items in the series things related to brick killings (e.g. 0.5 the brick is thrown at your head and kills you, 0.25 you trip over it and then hit your head, 0.125 etc.) but without explaining a priori why these particular probabilities should be assigned to these events, we haven't demonstrated anything informative.
If we knew that the probabilities of AI killing us were e.g. 0.5 in 2040, 0.5 in 2041 if not before that, etc. then indeed we'd know that p(doom) was essentially 1, just as p(brick_death) would be 1 if we knew the probabilities to be 0.5, 0.25, etc.
But both of those would assume their conclusions, so they aren't useful excercises.
[Not only do we not have a particular reason to assign such probabilities, we have reason to doubt them. If, you have a gun riggen to a fair coin, and you'll be shot if the coin lands on heads, and the coin is flipped at regular intervals, then you'd get 0.5 + 0.5^2, etc. But if you have a gun rigged to a coin that may or may not be a fair coin, then the more flips go by without a heads, the less likely the coin is to be fair, so the probability of each subsequent flip being a heads conditioned on the previous ones being tails will be less than 0.5. AI doom, whose probability we don't know with certainty would be like the latter example.]
In contrast, many of Richard's arguments precluding AI doom can be true at once. As their number increases, the probability automatically approaches 0. The probability can still be high if you think each argument is bad, but probably not as high as you might initially think by conisdering the average strength of arguments, rather than cumulative power.
To repeat, this is not the case with the many ways AI can cause doom. An infinite number of potential ways that AI could kill us doesn't mean that the probability is 1.
A series of fractions being infinitely long doesn't necessitate it converging on 1. For example, the series 1/2 + 1/8 + 1/32 ... converges on 2/3
True. For that to be the case, there needs to be a win condition, not just a loss condition.
> a random brick killing you. There are probably an infinite number of ways that could occur, but that doesn't mean that the probability is 1.
And the brick killing me is unlikely because there are so many other things that could kill me first. (Plus some chance of becoming an immortal uploaded mind?)
If our only options are DOOM and reroll the dice, we land on DOOM eventually. This is a simple coherent world model in which the probabilities sum to 1. (I mean there are things where DOOM gets substantially less likely each year where P(doom ) doesn't converge to 1).
So. Imagine we have somehow survived a million years without AI doom. You need the probability of doom in the next year to be Extremely small.
> But if you have a gun rigged to a coin that may or may not be a fair coin, then the more flips go by without a heads, the less likely the coin is to be fair, so the probability of each subsequent flip being a heads conditioned on the previous ones being tails will be less than 0.5. AI doom, whose probability we don't know with certainty would be like the latter example.]
No real world probability is exactly 0 or 1.
But if we think unfair coins are very unlikely, then our probability is very close to 1.
> The probability can still be high if you think each argument is bad, but probably not as high as you might initially think by conisdering the average strength of arguments, rather than cumulative power.
Substituting quantity of argument for quality isn't a good move.
Plenty of religious people try the "I have 300 arguments that my god exists". If you have an unbiased source searching all arguments, and it keeps finding weak arguments for X, then sure. Use the quantity. But it is possible for a human to write many weak arguments for a position, even when that position is known false and quite silly.
I don't find that convincing because every other technology has been good at something humans suck at. For example, we are pound for pound one of the most physically weak creatures that exist. Put a human up against any other predator of it's same weight, and the human loses. We can't jump or climb trees or run nearly as fast as many other creatures, we can't work as tirelessly as other creatures, our balance is terrible and walking on two legs is equivalent to a four wheel drive vehicle versus a bicycle. Physically we're trash...chimps can kick our ass even if we outweigh them. Most technologies thus far have been so helpful to us because they provide the physical power we don't have and save us (and beasts of burden) from physical labor.
The other thing machines have been good at is doing calculations and memory...once again, not a great strength of humans (though admittedly better than most other creatures).
The ONE thing we have going for is is verbal and conceptual intelligence. It's the one area where we excel so far past every other species that whichever one is in second place is so distant as to not even be in the same contest. Our capacity for intelligence and all the calories and resources that go to our brains is presumably precisely why our evolution has led to such comparatively weak bodies, because it's irrelevant.
So when for the very first time a new technology takes the one thing we excel at, far beyond every organism we have knowledge has ever existed, and starts besting us...that is completely new, different, and inherently threatening things. We're literally racing to create our own competitors at the one thing that let us exercise total dominion over every other one of the millions of species on earth.
The AI risk topic used to get me worked up, but I've come around to accepting that there's no reason I should give a shit, since I don't truly care whether or not humanity exists in 150 years. But based on all the non-stop hand wringing about the "fertility crisis", it seems that lots of people really DO care (or think they do) about humanity continuing to exist in 100 or 1000 years. And for those people, I can't understand why they don't see the obvious threat. Seemingly their curiosity outweighs their emotional investment in hypothetical future humans.
I see the development of AI as a continuation of an evolutionary process in which systems of greater and greater complexity largely replace more primitive ones (biological evolution, evolution of human societies), as discussed in the book Nonzero. Human societies have not been around for long, and the change in them due to technology is radical and speeding up and it’s hard to predict the outcome in terms of human safety.
Evolution? Evolution is only applicable to biology. At the root of the evolutionary process is mutation through natural selection. There is no guarantee that mutation will lead to greater complexity but survival. Bacteria for example is not complex, as far as living things go, and yet it survives and mutates in numbers beyond comprehension. The idea of simple being replaced by complex would also render the biological world meaningless, vegetation is simple, but without vegetation, organisms that we consider highly complex would cease to exist. Simple doesn't give way for the complex it provides for its existence and sustains it.
But moving beyond biology If you were to apply evolution to societies like you suggested I think it is as C.S. Lewis put it Chronological Snobbery to believe that societies have evolved in any real sense of the word. Certainly, we have changed and with every knew technology we find ways to make life a little easier but at the end of the day we are doing the same kinds of things that the Mesopotamians did. We build homes, get married, fight wars, grow gardens, and sail the world. We write poetry, sing songs, and play instruments, just like every other civilization. There is no evolution of human society from one form to the other. The Greeks were democrats long before the French. Our main and really only difference between our ancestors and ourselves is that we have discovered a highly concentrated form of energy to move things more efficiently with. None of this proves to me that we are inevitably going to be replaced by something more complex. The only inevitability is that the Sun will one day consume the earth. Not that aliens, a.i. or monkeys will replace humans.
> That argument has to grapple with the fact that there have been a lot of technologies and none have ever killed us all or even really come close in recorded history.
There is so much speculation at hand. How can anyone make a correct perdication about even next year? We assumed that nuclear power was going to destroy us all. (For that matter, which is one technology that certainly could and might.) We had our first taste in over a hundred years of a pandemic, we flirt with world war, and we assume that the oil will just keep coming out of the ground. We assume that the digital world is superior to the analog and yet not every country is so quick to turn their grid to the digital world. How often are prophets true?
I really thought you would call them out for demagoguery. The steelman argument for regulation is that social conservatism was beneficial in terms of survival and reproductive fitness throughout all of human history. Many conservatives believe that the impact of tech on reproduction is negative. I am vexed that “rationalists” even consider such specific rhetoric as having > 1% probability of being true. Why not drop the sci-fi rhetoric and instead appeal to conservatism? I bet tech causes fertility decline is too mainstream and right-wing for them. Extinction via fertility decline is more likely than global genocide. The doomers all seem pretty liberal to me, so I wouldn’t doubt that their narratives are simply a projection of their own psychology.
> Extinction via fertility decline is more likely than global genocide.
That is an Interesting position.
Firstly, extinction via fertility decline won't be quick, like even with really low fertility, 1 child per woman, that's a population halving every 30 years. That's 600 years to get from 8 billion to 8 thousand people.
Now there are various fertility cults, breeding fetishists, people allergic to all forms of birth control and general pro-natalists. And we are talking about a lot of genetic and cultural evolution. Your going to get some random sect that thinks birth control is evil, and their population won't be going down.
Then well the conditions that discourage reproduction are highly modernity specific, they involve being in a high tech world with limited housing space in cities, expanding education costs etc. These are not conditions that apply to a tribe of 100 humans scavenging at the remains of our cities.
Human extinction from fertility decline is rather implausible.
My bad. I view the most productive members of society as almost a distinct species. I don’t see how different geographically isolated subgroups don’t make global genocide much more difficult as well unless you argue than an AI will just nuke everything.
I agree-- risk from AI is quite disjunctive. There are a lot of potential serious problems, and each is reasonably independent from the others. To elaborate, here are possible sub-problems that AI risk can be broken down into:
Steerability (Inner Alignment). It may be very difficult to orient AI systems towards any goal. This problem has three largely non-overlapping subcomponents:
- Misgeneralization
- The imperfect reward signal problem
- Scheming
If steerability/inner alignment is completely solved, there is another problem: Outer Alignment. We may not choose the correct goals to use our alignment techniques to cause AIs to pursue. Or there are factions of humanity that want to do things that are bad by everybody else's lights, and it may be easier for their advanced AIs to succeed in these bad goals than it is for the combined power of everyone else to defend. So, outer alignment decomposes into:
- Misuse & Attack-Defense Imbalances
- Correctly aggregating human values
Finally, even if both of the above are solved, there remains problems with selection pressures. (side note, I think risks from selection pressures have received far too little attention). There will be a resource and power overhang if everybody's AIs are aligned to human interests, and fundamental forces of evolution favor the systems that leverage them to replicate and outcompete competitors. These forces exert tremendous pressure to disrupt the unstable equilibrium of a multipolar world of aligned AIs.
So yeah there are good reasons to reject the idea that doomers stand atop a rickety tower of assumptions. The problem is far more of a hydra than that.
P(extinction) = 1 - p(no extinction) = 1 - sum_i{ p(no extinction due to i) } > 0
(I have assumed independence in the second to last step)
It's still a positive number and AI doomers will then point out that extinction is really bad and even a small probability is a huge concern.
IMO the bigger issue is simply that AGI is really just not that close. Progress on text and image problems has been speedy over the last decade or so, but not so much for robots interacting with the physical world. The learning paradigm today (multi-layer nets plus SGD) is crazy inefficient compared to e.g.. how humans learn things. That's OK for digital data, but a huge obstacle to learning from physical reality, where we can't straightforwardly scale up the processing of training examples.
I don't think we need to worry about AI risk until we see substantial progress on the key challenges in achieving intelligence outside of the purely digital world.
No anti doomer has claimed that extinction due to AI has no chance of ever happening. We are simply against Terminator level rhetorical narratives that are pervasive within the EA culture (it’s not very rationalist). I really doubt that they hold the private position of conservatism in general and use stories as a Machiavellian approach. We simply want to get them to admit that regulation is a net negative under our current understanding of AI (the transformer architecture).
> I don't think we need to worry about AI risk until we see substantial progress on the key challenges in achieving intelligence outside of the purely digital world.
That is unlikely to leave you worrying for very long. Suppose that this robotics progress reliably appeared at least 2 year before ASI in every world. So what? We haven't seen it yet. That just means (if there isn't a secret manhatten project) that we have at least 2 years. 2 years isn't that long to solve alignment.
> AI doomers will then point out that extinction is really bad and even a small probability is a huge concern.
Straw man. Or at least weak man. If you have a convincing argument that P(doom)<10%, I want to see it.
You don't. When you screw around with basic probability theory, and get an estimate close to 0 or 1 without strong real world data, usually you are making an unwarranted independence assumption.
> IMO the bigger issue is simply that AGI is really just not that close.
> The learning paradigm today (multi-layer nets plus SGD) is crazy inefficient compared to e.g.. how humans learn things.
True. How much of a change would make that stop being true? Maybe we are 1 transformer sized breakthrough from efficient nets. And all the hardware and software and training data is already there. 20 lines of code and a day training and there is the superintelligence.
That’s not what the AI doomers are even arguing. They all seem to parrot either surreptitious collusion among AGIs to wipe out humanity to maximize utility or the paperclip maximizer argument. Those arguments are really just a form of astrology for males. I don’t think there is much to be done from a regulation standpoint for adversaries with strong manufacturing capabilities (China). Why not focus on the three letter agencies instead? We have good priors that they may be interested in using AI to subjugate us thanks to Snowden.
For the centralization of power into a single AGI argument, we already have good evidence against that with multiple companies having LLMs of similar levels of ability. Regulations will make centralization self-fulfilling. Why isn’t the rational take to leave innovators alone?
On Zeno’s Paradox. I was once discussing this in a college philosophy class. My professor challenged the class to come up with a compelling response. I tore a piece of paper out of my notebook, wadded it into a ball, and threw it at the Professor. He called me a “crude empiricist.”
The core argument for Doomerism doesn't rest on most of these assumptions at all. It's much more basic: if we create another intelligent species, that is significantly smarter than us, and reproduces faster than us what happens to us is no longer within our control. The doom here is not that humans get exterminated in some Terminator scenario, it's that we become pets/zoo objets/creatures confined to a nature preserve, while AI takes over the universe. And the only way that doesn't happen is if there is some fundamental magic preventing the AIs from being significantly more capable than humans. Everyone wants to analogize AI to technology, but that analogy *assumes away* exactly the thing that concerns people about AI.
Of these eight arguments, only 7 isn't one where the doomers won't immediately come after you with a mix of laugh and specific arguments why probabilities of all of these are infinitesimal. And their estimate for 7 is likely to be much lower, too (partly due to "accounting for unknown unknowns", i.e. discounting due to possible paradigm shifts).
(In particular, 3 is denying "instrumental convergence": nearly any goal will find useful more influence as a subgoal, and someone who thinks "women" - or, at least, stereotypical women - are an exception to evolving this subgoal hasn't much contacted with them in situations where they want something; 5 is "human vs. ant"; and so on and so forth.)
This is not to say that the argument is necessarily wrong, just that none of these have been missed by the doomers.
Spot on, except the mix contains also 'it's all so tiresome' in addition to laughter and arguments.
Yeah, a woman isn't a conquistador -- I'm sure the monkeys facing habitat destruction so one can buy a big tub of organic coconut oil at costco cares a lot about that.
yes, and women are part of a dyadic system, evolved in concert with the stabby conquistadors. if they're less violent it may be in part because they've had the privilege of oursourcing that violence to their menfolk, then sharing in the benefits of that violence, including the benefit of engorged brainflesh.
yes but that belongs to a different domain, doesn't really address the AI-specific mistake in reasoning that underlies the analogy in the first place. Any specific scenario about AI doom will look like science fiction (because it involves AI), including one in which different AIs team up -- in which case only one has to be a conquistador in order to destroy humanity. "haha silly dystopian scenarios where AIs collaborate more with one another than with us".
The point is that nobody even has to be killing the monkeys -- they just have to want a product that grows where the habitat used to be for the monkeys to starve and die. The same women buying the coconut oil or palm kernel oil or organic cacao or whatever will shed a tear when a tiktok of the last monkey of its species dying in captivity passes by on their screens, and then that species will be no more.
One potential issue with this style of analysis is that you could run it the other way and consider the various paths to some kind of AI doom/catastrophe. Then you might get a lot of small probabilities adding up to a large number.
I’m not sure how to resolve the tension between these two styles of reasoning.
That’s why this post is stupid. You can NOT reason this way and get useful answers.
If you did the same for doomer beliefs you’d end up with a number other than 96% even though mirrored statements necessarily add up to 100%. Ergo bad reasoning.
Also god damn these probabilities are obviously dependent. The 40% he tacks on at the end is blatantly so.
> I think a 4% chance that AI wipes us out and we can do something about it is high enough that people should work on the problem, and I’m glad some do.
I think that's compatible with the AI doomerism view, so it doesn't seem like you actually disagree with their core point? Once there is agreement on this core point, we can then quibble on whether slowing down AI research is a viable solution etc., but I think opening your post with a claim that you disagree with AI Doomerism is misleading. It's not necessary for the probability of the extinction of the human race to be "high" (whether that means to you something like "more than 50%" or "more than 0.1%" or whatever) for the AI Doomerism argument to be compelling.
My issue with the main crux of the article is that all those probabilities are likely to be correlated, you can't just multiply them all by each other, that only works when the things in question are wholly independent. You could make a pro-doomer argument along similar lines by racking up pro-doomer arguments
A smaller correction
" it’s difficult to think of many technologies that people have voluntarily used that did not make the world better off in the long run"
This is simply survivorship bias. I'm sure there is a random technology invented in the 1800s that just didn't so everyone forgot about it. You also can't assume that the technology in question will be imposed only voluntarily. If someone creates an AI capable of self-improvement, and in fact does so while being misaligned, despite me never agreeing to be affected by it, its need for resources will affect me
" it’s difficult to think of many technologies that people have voluntarily used that did not make the world better off in the long run"
Nearly every weapon.
CFC's. Asbestos. Leaded petrol. Cigarettes and other addictive drugs.
In most of those cases, the people taking the harm aren't the people using it. Ie leaded petrol poisoning everyone in the town a tiny bit, not just the people who use it. Your benefit, their problem.
I agree that anyone > 50% is probably thinking about this incorrectly.
But I disagree that 4% risk is not vitally important, because all of that risk is probably spread over the next 10 years. It's the single most important thing that will happen over that time.
I also think your entire system is backwards. It's not "these are reasons why something won't happen" because you can just keep thinking of more and more and more and more until you end up at whatever arbitrarily low number you desire. You have to do it the other way around - these are why something COULD happen. You'll quickly run out of reasons that aren't straightforwardly stupid.
(I happen to think that the risk is more like 30% over the next 10 years: we're about to make something much smarter than us, it will have different goals to ours, we don't know how to change those goals, and it might achieve them. I think all of those statements are true to various probabilities, and I think we need all of them to be true for there to be enough risk)
Also, one should consider that the possible “doom” scenarios are much more varied than the ones commonly predicted by prominent AI “doomers” (i.e., one or few uncontrollable AIs taking over). And many of those are quite compatible with one or more of the “optimistic” assumptions being true.
In particular, even with reasonable controllable AIs, it’s not at all clear what chance of survival most of humanity would have in a world where increasingly many decisions are delegated to inscrutably super-intelligent AIs and where humans are no longer able to earn above-subsistence wages. (Yes I’m aware of the comparative advantage argument, but that still depends on the marginal cost of human-equivalent compute and robots somehow remaining higher than that of human labor, an assumption that seems dubious at best.)
4% risk of dying (that I can prevent) would be the ONLY thing I cared about if it affected me as an individual. Would you eat food that had a 4% chance of being lethal poison? Would you play Russian roulette with a gun with 1 bullet out of 25? Even setting aside the rest of humanity, a 4% chance of dying is HUGE. And if all of humanity is dying alongside me, I care even more than I would if it were only me at risk.
But isn't the problem that there might be other issues with higher probability of extinction that we should prioritise? E.g. climate change, pandemics etc
Your 4% chance is still _really high_ when we're talking about existential risk. It easily justifies spending billions on AI Safety research and a regulatory apparatus that gives us a near-guarantee it is existentially safe, at the cost of slowing down capabilities research.
I wouldn't get on a plane with a 4% chance of a crash. When the odds of existential risk from AI disaster are closer to the odds of my next flight crashing to the ground, I'll be happy hop aboard the full steam ahead AI train.
I've placed the odds at 5-10% myself for the last couple of years or so, and I also sign on to your 1-5% "risky but we can do something about it" assessment. So I agree with your overall conclusion about what the risk is. I just think that's an _obvious_ sign we need to be more cautious.
There are two real arguments: (1) an out-of-control AI superintelligence won't happen, and (2) if it does happen, it won't decide to kill us. I really think each of those is worth evaluating on its own merits, rather than spamming arguments on other side.
My bet is mostly on (1) because I think (2) is effectively unknowable. And the most likely path for (1) is this: AI-related investments stop paying off, because each improvement requires an exponential increase in resources. At some point companies like MSFT realize that the next major upgrade of AI will be 5-10% better for practical business purposes (which will see diminishing returns long before abstract measures of intelligence), but it will cost $1 trillion to build, and the payoff isn't worth it. And so no one bothers to build it.
Compare to something like manned space travel: it turns out that going into Earth's orbit is practical, for artificial satellites, space tourism, etc. After that, the productivity benefits of space travel drop off exponentially. We could send a manned mission to Mars, but it would cost more money than anyone wants to pay. So it never happens. If it ever does happen, it will be so incredibly costly and pointless that, just as we never went back to the moon, no one will want to try anything like it again for 50+ years.
Same for data acquisition. The LLMs are consuming hyper condensed thoughts and theories about the world that took thousands of years for all of humanity to discover. If it ever reaches the edge of human data it has to rival our data acquisition system to keep expanding.
Another worry is that as a non-technical person (as far as I know), it might be hard for you to assign a low probability on any of those individual statements which will inevitably make the total calculated probability of us being safe high. While it is much more likely that a technical person could assign a near zero probability to at least some of those statements.
The one thing I would take issue with is the idea that the smarter a person is the more moral. I bet Stalin had a pretty high IQ, as he read literature. Higher than the vest majority of peasants. But also very evil. This logic also applies to the Nazis and most dictatorships including the Islamic ones. So a super smart AI could not be counted on to be moral. It could, like Stalin, reason that certain classes or person were just too much in the way of a utopia.
Some people have already come up with (relatively) optimistic estimates of AI existential risk with a similar method of multiplying probabilities of several necessary conditions.
Notably, there was a study commissioned by an EA organization, which also came up with a single-digit percentage probability (although the author notes that he has updated towards a higher estimate in the meantime):
Yudkowsky and other pessimists have criticized this kind of reasoning as “multiple stage fallacy”, where you end up multiplying probabilities of each stage as independent events, whereas in reality it would require an extremely difficult estimate of conditional probabilities.
I think Yudkowsky originally coined this term in a different context years ago, in response to a 2015 article by Nate Silver that used similar reasoning to estimate an extremely low (2%) probability for Trump to win the Republican nomination:
In retrospect it seems clear that Silver’s reasoning was fallacious exactly along these lines, leading to a severe underestimate of the final probability. (And that’s from someone whose grasp of probability is generally first-rate!) This, I think, should make one very skeptical about analogous optimistic estimates of AI risk.
The argument is a fundamentally flawed way of using probabilities, a la Zeno’s paradox. One could also argue that we are doomed by AI because there are multiple ways AI could kill us, each with a probability, we are dead if only one of them is true, and there’s only an infinitesimal chance all the dice rolls come out in our favor.
That argument has to grapple with the fact that there have been a lot of technologies and none have ever killed us all or even really come close in recorded history. In fact, they’ve basically all been good. If you have two arguments where both seem equally plausible to you and one would have been correct at every point in human history and the other would have been wrong, which do you go with?
that's just survivorship bias. If we invented a technology capable of wiping us out, then we wouldn't be around to discuss that technology has in the past wiped us out
We did invent a technology capable of wiping us out: nuclear weapons.
This seems to be besides the point? Your argument can be made for anything. Therefore it gives no information. You can introduce a new argument about the burden of proof or something, but the original post doesn't make any sense.
You're right, of course, but only due to #7 (which I'd put at 98.5%). The other arguments are flawed and Yudkowsky has debunked all of them at least once, as he has also this last argument.
No, creating something smarter than us, even via technology, isn't at all like other technology we've created. Our common ancestor with chimps created termite fishing and human intelligence both. One of these is not like the other.
The difference is that we have all the power in the world to turn off the machine and go outside.
You can’t just “turn it off”. You’re assuming it will have a green LED that turns red if it decides to be evil. Humanity won’t know anything is amiss until the AI has already wiped us out.
No, he's onto something there. Literally nobody has ever thought of this before. You are witnessing cutting edge alignment research.
I was speaking metaphorically.
I think the argument you're responsing to is incorrect, but for a different reason than your response. Your intitial reasoning suggests a low p(doom). The responding comment suggests a high p(doom). You can't simply resort to the null hypothesis - that ignores the fact that ostensibly equivalent reasoning is yielding opposite conclusions. It isn't a matter of two different arguments that sound equally compelling, but ostensibly, of the same reasoning yielding contradictory results - a paradox that must be resolved.
The resolution, I think, is that in your argument each probability can be arbitrarily low. In contrast, the probabilities of the different ways AI doom can occur are bounded, since they have to sum to less than 1, so they can't be arbitrarily high.
To restate that, in your initial argument, for AI doom to occur, a series of independent possibilities all must be incorrect. Each additional reason drastically lowers p(doom). There's no constraint stopping p(doom) from approaching 0; as the number of arguments accrue, the probability naturally approaches 0.
However, in looking at the probabilities of the various ways AI doom can occur, each probability must be tiny, since they're mutually exclusive. And since they therefore aren't independent, there's no reason to think that additional possibilities would lead to convergence on 1.
[That's why the comparison to Zeno's paradox is incorrect. In Zeno's paradox, an object asymptotically approaches a destination, but in adding all the possible variations of AI doom, we don't asymptotically approach 1 - we just asymptotically approach p(doom) - but that doesn't doesn't tell us anything about what p(doom) is.]
Regarding the argument that we should assume AI will be harmless since previous technologies have been harmless, the big question is the appropriate reference frame, as noted by Scott Alexander here (https://www.astralcodexten.com/p/mr-tries-the-safe-uncertainty-fallacy):
Suppose astronomers spotted a 100-mile long alien starship approaching Earth. Surely this counts as a radically uncertain situation if anything does; we have absolutely no idea what could happen. Therefore - the alien starship definitely won’t kill us and it’s not worth worrying? Seems wrong.
What’s the base rate for alien starships approaching Earth killing humanity? We don’t have a base rate, because we’ve never been in this situation before. What is the base rate for developing above-human-level AI killing humanity? We don’t . . . you get the picture.
You can try to fish for something sort of like a base rate: “There have been a hundred major inventions since agriculture, and none of them killed humanity, so the base rate for major inventions killing everyone is about 0%”.
But I can counterargue: “There have been about a dozen times a sapient species has created a more intelligent successor species: australopithecus → homo habilis, homo habilis → homo erectus, etc - and in each case, the successor species has wiped out its predecessor. So the base rate for more intelligent successor species killing everyone is about 100%”.
The Less Wrongers call this game “reference class tennis”, and insist that the only winning move is not to play. Thinking about this question in terms of base rates is just as hard as thinking of it any other way, and would require arguments for why one base rate is better than another.
That was a very intelligent comment and a pleasure to read.
Also I hadn't encountered the base rate argument in terms of wiping predecessor species. That seems worth looking into in greater detail -- why exactly do the ancestors disappear when the successor species become dominant?
Thanks! Hopefully someone familiar with hominid evolution and history can elaborate on that.
> However, in looking at the probabilities of the various ways AI doom can occur, each probability must be tiny, since they're mutually exclusive. And since they therefore aren't independent, there's no reason to think that additional possibilities would lead to convergence on 1.
Suppose there is a 50% chance of AI killing us in 2040, and if it doesn't, there is a 50% chance of it killing us in 2041 and so on for all years into the future.
That looks like converging to 1 to me.
I long list of "If the previous problem hasn't already killed you, this one is likely to" can converge to 1.
A series of fractions being infinitely long doesn't necessitate it converging on 1. For example, the series 1/2 + 1/8 + 1/32 ... converges on 2/3.
The same is true for probabilities. 1 is the most probabilites of an event can sum to, but that doesn't mean that all events have a probability of 1, even if there are infinitely many ways they can occur.
We can illustrate this with probabilities by considering an event we consider very unlikely, e.g. a random brick killing you. There are probably an infinite number of ways that could occur, but that doesn't mean that the probability is 1.
Richard's point is that arguements that preclude AI doom are largely independent, so each needs to be incorrect for doom to be possible (not just for doom to occur, incidentally; that is, they form an upper bound for p(doom), not an estimate for it), so as the number of arguments accrue, p(doom) appoaches 0.
This isn't just a semantic game that can be applies to any values or any probabilities. As the series demonstrated, not all series converge on 1. And as the brick example illustrated, an event, however unlikely, can have its probability space partitioned infinitely, but that doesn't make the event more likely, and it certainly doesn't give it a probability of 1.
One could give an example of a series of probabilities that converges to 1, and then label the items in the series things related to brick killings (e.g. 0.5 the brick is thrown at your head and kills you, 0.25 you trip over it and then hit your head, 0.125 etc.) but without explaining a priori why these particular probabilities should be assigned to these events, we haven't demonstrated anything informative.
If we knew that the probabilities of AI killing us were e.g. 0.5 in 2040, 0.5 in 2041 if not before that, etc. then indeed we'd know that p(doom) was essentially 1, just as p(brick_death) would be 1 if we knew the probabilities to be 0.5, 0.25, etc.
But both of those would assume their conclusions, so they aren't useful excercises.
[Not only do we not have a particular reason to assign such probabilities, we have reason to doubt them. If, you have a gun riggen to a fair coin, and you'll be shot if the coin lands on heads, and the coin is flipped at regular intervals, then you'd get 0.5 + 0.5^2, etc. But if you have a gun rigged to a coin that may or may not be a fair coin, then the more flips go by without a heads, the less likely the coin is to be fair, so the probability of each subsequent flip being a heads conditioned on the previous ones being tails will be less than 0.5. AI doom, whose probability we don't know with certainty would be like the latter example.]
In contrast, many of Richard's arguments precluding AI doom can be true at once. As their number increases, the probability automatically approaches 0. The probability can still be high if you think each argument is bad, but probably not as high as you might initially think by conisdering the average strength of arguments, rather than cumulative power.
To repeat, this is not the case with the many ways AI can cause doom. An infinite number of potential ways that AI could kill us doesn't mean that the probability is 1.
A series of fractions being infinitely long doesn't necessitate it converging on 1. For example, the series 1/2 + 1/8 + 1/32 ... converges on 2/3
True. For that to be the case, there needs to be a win condition, not just a loss condition.
> a random brick killing you. There are probably an infinite number of ways that could occur, but that doesn't mean that the probability is 1.
And the brick killing me is unlikely because there are so many other things that could kill me first. (Plus some chance of becoming an immortal uploaded mind?)
If our only options are DOOM and reroll the dice, we land on DOOM eventually. This is a simple coherent world model in which the probabilities sum to 1. (I mean there are things where DOOM gets substantially less likely each year where P(doom ) doesn't converge to 1).
So. Imagine we have somehow survived a million years without AI doom. You need the probability of doom in the next year to be Extremely small.
> But if you have a gun rigged to a coin that may or may not be a fair coin, then the more flips go by without a heads, the less likely the coin is to be fair, so the probability of each subsequent flip being a heads conditioned on the previous ones being tails will be less than 0.5. AI doom, whose probability we don't know with certainty would be like the latter example.]
No real world probability is exactly 0 or 1.
But if we think unfair coins are very unlikely, then our probability is very close to 1.
> The probability can still be high if you think each argument is bad, but probably not as high as you might initially think by conisdering the average strength of arguments, rather than cumulative power.
Substituting quantity of argument for quality isn't a good move.
Plenty of religious people try the "I have 300 arguments that my god exists". If you have an unbiased source searching all arguments, and it keeps finding weak arguments for X, then sure. Use the quantity. But it is possible for a human to write many weak arguments for a position, even when that position is known false and quite silly.
I don't find that convincing because every other technology has been good at something humans suck at. For example, we are pound for pound one of the most physically weak creatures that exist. Put a human up against any other predator of it's same weight, and the human loses. We can't jump or climb trees or run nearly as fast as many other creatures, we can't work as tirelessly as other creatures, our balance is terrible and walking on two legs is equivalent to a four wheel drive vehicle versus a bicycle. Physically we're trash...chimps can kick our ass even if we outweigh them. Most technologies thus far have been so helpful to us because they provide the physical power we don't have and save us (and beasts of burden) from physical labor.
The other thing machines have been good at is doing calculations and memory...once again, not a great strength of humans (though admittedly better than most other creatures).
The ONE thing we have going for is is verbal and conceptual intelligence. It's the one area where we excel so far past every other species that whichever one is in second place is so distant as to not even be in the same contest. Our capacity for intelligence and all the calories and resources that go to our brains is presumably precisely why our evolution has led to such comparatively weak bodies, because it's irrelevant.
So when for the very first time a new technology takes the one thing we excel at, far beyond every organism we have knowledge has ever existed, and starts besting us...that is completely new, different, and inherently threatening things. We're literally racing to create our own competitors at the one thing that let us exercise total dominion over every other one of the millions of species on earth.
The AI risk topic used to get me worked up, but I've come around to accepting that there's no reason I should give a shit, since I don't truly care whether or not humanity exists in 150 years. But based on all the non-stop hand wringing about the "fertility crisis", it seems that lots of people really DO care (or think they do) about humanity continuing to exist in 100 or 1000 years. And for those people, I can't understand why they don't see the obvious threat. Seemingly their curiosity outweighs their emotional investment in hypothetical future humans.
> there have been a lot of technologies and none have ever killed us all or even really come close in recorded history
None of those technologies has been super-human intelligence. it really is a game changer!
I see the development of AI as a continuation of an evolutionary process in which systems of greater and greater complexity largely replace more primitive ones (biological evolution, evolution of human societies), as discussed in the book Nonzero. Human societies have not been around for long, and the change in them due to technology is radical and speeding up and it’s hard to predict the outcome in terms of human safety.
Evolution? Evolution is only applicable to biology. At the root of the evolutionary process is mutation through natural selection. There is no guarantee that mutation will lead to greater complexity but survival. Bacteria for example is not complex, as far as living things go, and yet it survives and mutates in numbers beyond comprehension. The idea of simple being replaced by complex would also render the biological world meaningless, vegetation is simple, but without vegetation, organisms that we consider highly complex would cease to exist. Simple doesn't give way for the complex it provides for its existence and sustains it.
But moving beyond biology If you were to apply evolution to societies like you suggested I think it is as C.S. Lewis put it Chronological Snobbery to believe that societies have evolved in any real sense of the word. Certainly, we have changed and with every knew technology we find ways to make life a little easier but at the end of the day we are doing the same kinds of things that the Mesopotamians did. We build homes, get married, fight wars, grow gardens, and sail the world. We write poetry, sing songs, and play instruments, just like every other civilization. There is no evolution of human society from one form to the other. The Greeks were democrats long before the French. Our main and really only difference between our ancestors and ourselves is that we have discovered a highly concentrated form of energy to move things more efficiently with. None of this proves to me that we are inevitably going to be replaced by something more complex. The only inevitability is that the Sun will one day consume the earth. Not that aliens, a.i. or monkeys will replace humans.
> That argument has to grapple with the fact that there have been a lot of technologies and none have ever killed us all or even really come close in recorded history.
Irrelevant.
See the anthropic shadow effect. https://forum.effectivealtruism.org/topics/anthropic-shadow
There is so much speculation at hand. How can anyone make a correct perdication about even next year? We assumed that nuclear power was going to destroy us all. (For that matter, which is one technology that certainly could and might.) We had our first taste in over a hundred years of a pandemic, we flirt with world war, and we assume that the oil will just keep coming out of the ground. We assume that the digital world is superior to the analog and yet not every country is so quick to turn their grid to the digital world. How often are prophets true?
I really thought you would call them out for demagoguery. The steelman argument for regulation is that social conservatism was beneficial in terms of survival and reproductive fitness throughout all of human history. Many conservatives believe that the impact of tech on reproduction is negative. I am vexed that “rationalists” even consider such specific rhetoric as having > 1% probability of being true. Why not drop the sci-fi rhetoric and instead appeal to conservatism? I bet tech causes fertility decline is too mainstream and right-wing for them. Extinction via fertility decline is more likely than global genocide. The doomers all seem pretty liberal to me, so I wouldn’t doubt that their narratives are simply a projection of their own psychology.
> Extinction via fertility decline is more likely than global genocide.
That is an Interesting position.
Firstly, extinction via fertility decline won't be quick, like even with really low fertility, 1 child per woman, that's a population halving every 30 years. That's 600 years to get from 8 billion to 8 thousand people.
Now there are various fertility cults, breeding fetishists, people allergic to all forms of birth control and general pro-natalists. And we are talking about a lot of genetic and cultural evolution. Your going to get some random sect that thinks birth control is evil, and their population won't be going down.
Then well the conditions that discourage reproduction are highly modernity specific, they involve being in a high tech world with limited housing space in cities, expanding education costs etc. These are not conditions that apply to a tribe of 100 humans scavenging at the remains of our cities.
Human extinction from fertility decline is rather implausible.
My bad. I view the most productive members of society as almost a distinct species. I don’t see how different geographically isolated subgroups don’t make global genocide much more difficult as well unless you argue than an AI will just nuke everything.
I agree-- risk from AI is quite disjunctive. There are a lot of potential serious problems, and each is reasonably independent from the others. To elaborate, here are possible sub-problems that AI risk can be broken down into:
Steerability (Inner Alignment). It may be very difficult to orient AI systems towards any goal. This problem has three largely non-overlapping subcomponents:
- Misgeneralization
- The imperfect reward signal problem
- Scheming
If steerability/inner alignment is completely solved, there is another problem: Outer Alignment. We may not choose the correct goals to use our alignment techniques to cause AIs to pursue. Or there are factions of humanity that want to do things that are bad by everybody else's lights, and it may be easier for their advanced AIs to succeed in these bad goals than it is for the combined power of everyone else to defend. So, outer alignment decomposes into:
- Misuse & Attack-Defense Imbalances
- Correctly aggregating human values
Finally, even if both of the above are solved, there remains problems with selection pressures. (side note, I think risks from selection pressures have received far too little attention). There will be a resource and power overhang if everybody's AIs are aligned to human interests, and fundamental forces of evolution favor the systems that leverage them to replicate and outcompete competitors. These forces exert tremendous pressure to disrupt the unstable equilibrium of a multipolar world of aligned AIs.
So yeah there are good reasons to reject the idea that doomers stand atop a rickety tower of assumptions. The problem is far more of a hydra than that.
If most of the AI's are aligned, the cooperating superintelligences squash evolution flat.
Not sure exactly how. They are smart enough, they can figure it out.
I agree with Jim here.
P(extinction) = 1 - p(no extinction) = 1 - sum_i{ p(no extinction due to i) } > 0
(I have assumed independence in the second to last step)
It's still a positive number and AI doomers will then point out that extinction is really bad and even a small probability is a huge concern.
IMO the bigger issue is simply that AGI is really just not that close. Progress on text and image problems has been speedy over the last decade or so, but not so much for robots interacting with the physical world. The learning paradigm today (multi-layer nets plus SGD) is crazy inefficient compared to e.g.. how humans learn things. That's OK for digital data, but a huge obstacle to learning from physical reality, where we can't straightforwardly scale up the processing of training examples.
I don't think we need to worry about AI risk until we see substantial progress on the key challenges in achieving intelligence outside of the purely digital world.
No anti doomer has claimed that extinction due to AI has no chance of ever happening. We are simply against Terminator level rhetorical narratives that are pervasive within the EA culture (it’s not very rationalist). I really doubt that they hold the private position of conservatism in general and use stories as a Machiavellian approach. We simply want to get them to admit that regulation is a net negative under our current understanding of AI (the transformer architecture).
> I don't think we need to worry about AI risk until we see substantial progress on the key challenges in achieving intelligence outside of the purely digital world.
That is unlikely to leave you worrying for very long. Suppose that this robotics progress reliably appeared at least 2 year before ASI in every world. So what? We haven't seen it yet. That just means (if there isn't a secret manhatten project) that we have at least 2 years. 2 years isn't that long to solve alignment.
> AI doomers will then point out that extinction is really bad and even a small probability is a huge concern.
Straw man. Or at least weak man. If you have a convincing argument that P(doom)<10%, I want to see it.
You don't. When you screw around with basic probability theory, and get an estimate close to 0 or 1 without strong real world data, usually you are making an unwarranted independence assumption.
> IMO the bigger issue is simply that AGI is really just not that close.
> The learning paradigm today (multi-layer nets plus SGD) is crazy inefficient compared to e.g.. how humans learn things.
True. How much of a change would make that stop being true? Maybe we are 1 transformer sized breakthrough from efficient nets. And all the hardware and software and training data is already there. 20 lines of code and a day training and there is the superintelligence.
That’s not what the AI doomers are even arguing. They all seem to parrot either surreptitious collusion among AGIs to wipe out humanity to maximize utility or the paperclip maximizer argument. Those arguments are really just a form of astrology for males. I don’t think there is much to be done from a regulation standpoint for adversaries with strong manufacturing capabilities (China). Why not focus on the three letter agencies instead? We have good priors that they may be interested in using AI to subjugate us thanks to Snowden.
For the centralization of power into a single AGI argument, we already have good evidence against that with multiple companies having LLMs of similar levels of ability. Regulations will make centralization self-fulfilling. Why isn’t the rational take to leave innovators alone?
On Zeno’s Paradox. I was once discussing this in a college philosophy class. My professor challenged the class to come up with a compelling response. I tore a piece of paper out of my notebook, wadded it into a ball, and threw it at the Professor. He called me a “crude empiricist.”
The future will be science-fictional either way, we're just arguing over the subgenre.
The core argument for Doomerism doesn't rest on most of these assumptions at all. It's much more basic: if we create another intelligent species, that is significantly smarter than us, and reproduces faster than us what happens to us is no longer within our control. The doom here is not that humans get exterminated in some Terminator scenario, it's that we become pets/zoo objets/creatures confined to a nature preserve, while AI takes over the universe. And the only way that doesn't happen is if there is some fundamental magic preventing the AIs from being significantly more capable than humans. Everyone wants to analogize AI to technology, but that analogy *assumes away* exactly the thing that concerns people about AI.
Of these eight arguments, only 7 isn't one where the doomers won't immediately come after you with a mix of laugh and specific arguments why probabilities of all of these are infinitesimal. And their estimate for 7 is likely to be much lower, too (partly due to "accounting for unknown unknowns", i.e. discounting due to possible paradigm shifts).
(In particular, 3 is denying "instrumental convergence": nearly any goal will find useful more influence as a subgoal, and someone who thinks "women" - or, at least, stereotypical women - are an exception to evolving this subgoal hasn't much contacted with them in situations where they want something; 5 is "human vs. ant"; and so on and so forth.)
This is not to say that the argument is necessarily wrong, just that none of these have been missed by the doomers.
Spot on, except the mix contains also 'it's all so tiresome' in addition to laughter and arguments.
Yeah, a woman isn't a conquistador -- I'm sure the monkeys facing habitat destruction so one can buy a big tub of organic coconut oil at costco cares a lot about that.
yes, and women are part of a dyadic system, evolved in concert with the stabby conquistadors. if they're less violent it may be in part because they've had the privilege of oursourcing that violence to their menfolk, then sharing in the benefits of that violence, including the benefit of engorged brainflesh.
yes but that belongs to a different domain, doesn't really address the AI-specific mistake in reasoning that underlies the analogy in the first place. Any specific scenario about AI doom will look like science fiction (because it involves AI), including one in which different AIs team up -- in which case only one has to be a conquistador in order to destroy humanity. "haha silly dystopian scenarios where AIs collaborate more with one another than with us".
The point is that nobody even has to be killing the monkeys -- they just have to want a product that grows where the habitat used to be for the monkeys to starve and die. The same women buying the coconut oil or palm kernel oil or organic cacao or whatever will shed a tear when a tiktok of the last monkey of its species dying in captivity passes by on their screens, and then that species will be no more.
One potential issue with this style of analysis is that you could run it the other way and consider the various paths to some kind of AI doom/catastrophe. Then you might get a lot of small probabilities adding up to a large number.
I’m not sure how to resolve the tension between these two styles of reasoning.
That’s why this post is stupid. You can NOT reason this way and get useful answers.
If you did the same for doomer beliefs you’d end up with a number other than 96% even though mirrored statements necessarily add up to 100%. Ergo bad reasoning.
Also god damn these probabilities are obviously dependent. The 40% he tacks on at the end is blatantly so.
You conclude:
> I think a 4% chance that AI wipes us out and we can do something about it is high enough that people should work on the problem, and I’m glad some do.
I think that's compatible with the AI doomerism view, so it doesn't seem like you actually disagree with their core point? Once there is agreement on this core point, we can then quibble on whether slowing down AI research is a viable solution etc., but I think opening your post with a claim that you disagree with AI Doomerism is misleading. It's not necessary for the probability of the extinction of the human race to be "high" (whether that means to you something like "more than 50%" or "more than 0.1%" or whatever) for the AI Doomerism argument to be compelling.
My issue with the main crux of the article is that all those probabilities are likely to be correlated, you can't just multiply them all by each other, that only works when the things in question are wholly independent. You could make a pro-doomer argument along similar lines by racking up pro-doomer arguments
A smaller correction
" it’s difficult to think of many technologies that people have voluntarily used that did not make the world better off in the long run"
This is simply survivorship bias. I'm sure there is a random technology invented in the 1800s that just didn't so everyone forgot about it. You also can't assume that the technology in question will be imposed only voluntarily. If someone creates an AI capable of self-improvement, and in fact does so while being misaligned, despite me never agreeing to be affected by it, its need for resources will affect me
" it’s difficult to think of many technologies that people have voluntarily used that did not make the world better off in the long run"
Nearly every weapon.
CFC's. Asbestos. Leaded petrol. Cigarettes and other addictive drugs.
In most of those cases, the people taking the harm aren't the people using it. Ie leaded petrol poisoning everyone in the town a tiny bit, not just the people who use it. Your benefit, their problem.
I agree that anyone > 50% is probably thinking about this incorrectly.
But I disagree that 4% risk is not vitally important, because all of that risk is probably spread over the next 10 years. It's the single most important thing that will happen over that time.
I also think your entire system is backwards. It's not "these are reasons why something won't happen" because you can just keep thinking of more and more and more and more until you end up at whatever arbitrarily low number you desire. You have to do it the other way around - these are why something COULD happen. You'll quickly run out of reasons that aren't straightforwardly stupid.
(I happen to think that the risk is more like 30% over the next 10 years: we're about to make something much smarter than us, it will have different goals to ours, we don't know how to change those goals, and it might achieve them. I think all of those statements are true to various probabilities, and I think we need all of them to be true for there to be enough risk)
Also, one should consider that the possible “doom” scenarios are much more varied than the ones commonly predicted by prominent AI “doomers” (i.e., one or few uncontrollable AIs taking over). And many of those are quite compatible with one or more of the “optimistic” assumptions being true.
In particular, even with reasonable controllable AIs, it’s not at all clear what chance of survival most of humanity would have in a world where increasingly many decisions are delegated to inscrutably super-intelligent AIs and where humans are no longer able to earn above-subsistence wages. (Yes I’m aware of the comparative advantage argument, but that still depends on the marginal cost of human-equivalent compute and robots somehow remaining higher than that of human labor, an assumption that seems dubious at best.)
Here's a more rigorous version of this: it's not just about having longer series of probabilities, but trying to get them to be independence. https://www.strangeloopcanon.com/p/agi-strange-equation
4% risk of dying (that I can prevent) would be the ONLY thing I cared about if it affected me as an individual. Would you eat food that had a 4% chance of being lethal poison? Would you play Russian roulette with a gun with 1 bullet out of 25? Even setting aside the rest of humanity, a 4% chance of dying is HUGE. And if all of humanity is dying alongside me, I care even more than I would if it were only me at risk.
Would you ride in a car that crashes 4% of the time? Should the police allow that 96% safe driver to keep his license?
But isn't the problem that there might be other issues with higher probability of extinction that we should prioritise? E.g. climate change, pandemics etc
No, mitigating one does not preclude mitigating another. And neither of those other scenarios present a clear path to extinction like AI does.
Your 4% chance is still _really high_ when we're talking about existential risk. It easily justifies spending billions on AI Safety research and a regulatory apparatus that gives us a near-guarantee it is existentially safe, at the cost of slowing down capabilities research.
I wouldn't get on a plane with a 4% chance of a crash. When the odds of existential risk from AI disaster are closer to the odds of my next flight crashing to the ground, I'll be happy hop aboard the full steam ahead AI train.
I've placed the odds at 5-10% myself for the last couple of years or so, and I also sign on to your 1-5% "risky but we can do something about it" assessment. So I agree with your overall conclusion about what the risk is. I just think that's an _obvious_ sign we need to be more cautious.
There are two real arguments: (1) an out-of-control AI superintelligence won't happen, and (2) if it does happen, it won't decide to kill us. I really think each of those is worth evaluating on its own merits, rather than spamming arguments on other side.
My bet is mostly on (1) because I think (2) is effectively unknowable. And the most likely path for (1) is this: AI-related investments stop paying off, because each improvement requires an exponential increase in resources. At some point companies like MSFT realize that the next major upgrade of AI will be 5-10% better for practical business purposes (which will see diminishing returns long before abstract measures of intelligence), but it will cost $1 trillion to build, and the payoff isn't worth it. And so no one bothers to build it.
Compare to something like manned space travel: it turns out that going into Earth's orbit is practical, for artificial satellites, space tourism, etc. After that, the productivity benefits of space travel drop off exponentially. We could send a manned mission to Mars, but it would cost more money than anyone wants to pay. So it never happens. If it ever does happen, it will be so incredibly costly and pointless that, just as we never went back to the moon, no one will want to try anything like it again for 50+ years.
Same for data acquisition. The LLMs are consuming hyper condensed thoughts and theories about the world that took thousands of years for all of humanity to discover. If it ever reaches the edge of human data it has to rival our data acquisition system to keep expanding.
Another worry is that as a non-technical person (as far as I know), it might be hard for you to assign a low probability on any of those individual statements which will inevitably make the total calculated probability of us being safe high. While it is much more likely that a technical person could assign a near zero probability to at least some of those statements.
The one thing I would take issue with is the idea that the smarter a person is the more moral. I bet Stalin had a pretty high IQ, as he read literature. Higher than the vest majority of peasants. But also very evil. This logic also applies to the Nazis and most dictatorships including the Islamic ones. So a super smart AI could not be counted on to be moral. It could, like Stalin, reason that certain classes or person were just too much in the way of a utopia.
Some people have already come up with (relatively) optimistic estimates of AI existential risk with a similar method of multiplying probabilities of several necessary conditions.
Notably, there was a study commissioned by an EA organization, which also came up with a single-digit percentage probability (although the author notes that he has updated towards a higher estimate in the meantime):
https://arxiv.org/abs/2206.13353
Yudkowsky and other pessimists have criticized this kind of reasoning as “multiple stage fallacy”, where you end up multiplying probabilities of each stage as independent events, whereas in reality it would require an extremely difficult estimate of conditional probabilities.
I think Yudkowsky originally coined this term in a different context years ago, in response to a 2015 article by Nate Silver that used similar reasoning to estimate an extremely low (2%) probability for Trump to win the Republican nomination:
https://forum.effectivealtruism.org/posts/GgPrbxdWhyaDjks2m/the-multiple-stage-fallacy
In retrospect it seems clear that Silver’s reasoning was fallacious exactly along these lines, leading to a severe underestimate of the final probability. (And that’s from someone whose grasp of probability is generally first-rate!) This, I think, should make one very skeptical about analogous optimistic estimates of AI risk.