I think 2% and 17% chances of human extinction prompt very similar policy responses, specifically trying to lower the chances of it happening by a few orders of magnitude in either case.
I think it's a mistake to go "Well we should risk it because the upside is high". A lot of the non-extinction probability space is in AI just never becoming super human. If AI does become smart enough to cure all diseases and enable wide spread space colonization, then it's much more likely to be capable of killing us too. The more likely outcomes are either we die or not much changes. There aren't many scenarios where we're all helped by a merciful machine god.
I'm against unrestricted AI development and my p(doom | unrestricted AI development) is 80% but I disagree with this take. If my p(doom) was zero, I would be an extreme accelerationist. I actually can't envision how a friendly AI superintelligence _wouldn't_ basically overhaul what it means to be human (in a good way).
At every turn, every bottleneck to human advancement is blocked by intelligence and the application of intellectual effort.
If we never made another advance again and were permanently stuck with today's frontier models, we're still going to see an incredible technological transformation over the next decade or two like nothing you've ever experienced as general purpose robots are built out and become a fundamental part of everyday life, and all scientific efforts become essentially infinitely parallelizable. That's enabled by _today's_ frontier models becoming widespread and integrated into systems (In this scenario I'm assuming training smaller and narrow AI is still allowed, just not training new frontier general purpose models).
And it's important to understand this is the real reason I'm not gung ho about advancing _despite the risks_. We've already made massive strides in AI that aren't going to be rolled back. And p(immortality | no-superintelligence) is _also_ a decently high number. We don't need to incur a risk of extinction to get things we're going to get anyway
I should've been more clear. I think p(doom|superintelligence) is very high. I think p(superintelligence) is not necessarily that high. So p(doom) is not necessarily that high. But this does mean p(superintelligence & not-doom) is very low. So the upside scenario is very unlikely, and not something we should be gambling with doom to try to achieve.
Of course there's no hard numbers on this, but it's my impression from seeing nearly no one argue that both superintelligence and alignment are in our near future. Few have a vision of that
When you say you don't think p(superintelligence) is high, you're saying you think it's unlikely current techniques scale to superintelligence? Or that you think it's likely we will successfully identify where the danger threshold is and stop scaling before we get there (whether through policy or individual lab efforts)?
Some indeterminate share of optimistic forecasts surely prices in the possibility that this technology will be banned or otherwise seriously curbed by governments.
Inference along the lines "forecasters are not too worried about a huge disaster caused by AI, therefore we shouldn't regulate AI" is imho obviously invalid. Perhaps they are in fact predicting that disaster-preventing regulation is going to be enacted.
Yay! Thank you Richard for finally considering p(doom) seriously!
All of the numbers you use for your final estimate of between 2% and 17% are _marginal_ probabilities _given_ whatever coordinated safety efforts are predicted to the degree they succeed at slowing AI progress.
Please be careful not to treat them as like p(doom | no-safety-effort) which should be a higher number and the thing that drives policy decisions.
Safety effort is likely to make P(doom) less likely so even if we can't accurately calculate p(doom|safety-effort) or p(doom|no-safety-effort), it's clear that a safety effort moves things in the right direction.
Up until now Richard has been against AI regulation of any kind. His reasoning has been all the usual (usually correct) reasons for pushing back against Luddism. I'm glad to see him digging into the AI safety debate and asking whether this one really is different.
The whole discussion is polluted by people who think we're worried about job loss or copyright issues or water use as well as by people who (very stupidly IMO) actually _are_ worried about job loss and copyright issues and water use, and it's nice to see Richard firmly step out of that basin
I usually ask for their model of doom, not just a p(doom). Models reveal methods, can be analyzed, and debated. It’s why I take people who are worried about global warming seriously: They’ve modeled it, considered how variables interact, tested the parts they can, and then calibrated their models. My point isn’t a slam dunk, a model is not necessary or sufficient, but having a model separates serious people from those who just make up probabilities “that seem reasonable.” Blind empiricism is not the best path to knowledge. The next question doomers ask me is, “what would a model look like?” and I say, “That’s your job.”
The problem with outside view forecasting methods is the same problem as the AP arguments: how likely was a thing, which would have prevented us observing it if it occurred?
Given that we're here observing, it clearly didn't happen, but that says nothing about how likely it was.
The biggest point in favor of metaculus/rat forecasters to my mind has been their performance on AI up to this point - being consistently more correct than superforcasters about capabilities and behaviors.
The fundamental problem is that AI is completely and fundamentally unprecedented in human history. Forecasters have nothing to fall back on. That’s not to say no one knows anything, but it’s hard to assign confidence in anyone’s predictions, particularly if you’re relying on track record.
> If the payoff to superintelligence is that we cure most diseases and all live like billionaires, then where exactly your estimate falls within that range could determine which path you think we should take.
The chance of doom can be influenced by our actions, and it's not just a binary choice between stasis and a heedless rush to superintelligence. I think few people are advocating for never developing superintelligence at all. But let's say there is a 2% chance of doom if we rush (let's say we get there in 5 years then), and a 1% chance if we slow down and take 3 times as long (so 15 years). Some people will die of disease etc. in the extra time it takes to superintelligence, but on the plus side there is, in expectation, 1% of almost infinite future value.
Of course, this logic only works if you care about the wellbeing of future people - if you only care about those currently alive, you might say: "well more than 1% of people will die in those extra years". Then it becomes more important if your p(doom) is 2% or 15%. Is that the root of the disagreement?
There's a counterfactual where we don't develop superintelligence but _still_ cure most diseases and all live like billionaires, just get there a little bit slower. If not for that, I'd be way more willing to gamble
(Q1) How likely is it that humans will pause the development of strong AI, for maybe a decade or so, to find out how to align it better?
(Q2) if there is no pause, how likely is AI to kill us all?
(Q3) if there is a pause, how likely is AI to kill us all?
My answer to Q2 is at least 10% and possibly as high as 95%.
My answer to Q3 is i don't know but it is surely going to make AI doom significantly less likely.
Therefore I am for a pause.
Obviously the creation of a new life form more intelligent than humans has never happened before, so it is clearly an outside context problem, and very hard to calculate proabilities.
What you need to understand is that the only definite part of these estimates is that the chance is not 0, everything after that from specific estimates to ranges is almost entirely vibes and hyperactive imagination here.
Where this pretty clearly breaks down is that the best Metaculus usées _are_, by any reasonable definition, superforecasters themselves.
So we should take their views at least as much into account as the FRI superforecasters, and then consider if there are any reasons to believe one is more likely correct than the other.
My suspicion is that the Metaculus superforecasters are more likely correct than the FRI ones because we have much less information about the FRI one's level of ongoing engagement with forecasting than we do for the metaculus ones, but this is a very weak point. I don't have a stronger one so maybe a mean is appropriate here!
Stop repeating the current propaganda! The current propaganda states we have to build-out data centers to compete with China in the AI race. What nonsense. To use AI for legitimate purposes gigantic data centers are not required. The data centers are needed to surveil and control through Central Bank Digital currency and social credit scores EVERYONE in real time, all day every day, step by step! There is no need to compete with China on implementing totalitarianism unless your goal is totalitarianism.
I think 2% and 17% chances of human extinction prompt very similar policy responses, specifically trying to lower the chances of it happening by a few orders of magnitude in either case.
I think it's a mistake to go "Well we should risk it because the upside is high". A lot of the non-extinction probability space is in AI just never becoming super human. If AI does become smart enough to cure all diseases and enable wide spread space colonization, then it's much more likely to be capable of killing us too. The more likely outcomes are either we die or not much changes. There aren't many scenarios where we're all helped by a merciful machine god.
I'm against unrestricted AI development and my p(doom | unrestricted AI development) is 80% but I disagree with this take. If my p(doom) was zero, I would be an extreme accelerationist. I actually can't envision how a friendly AI superintelligence _wouldn't_ basically overhaul what it means to be human (in a good way).
At every turn, every bottleneck to human advancement is blocked by intelligence and the application of intellectual effort.
If we never made another advance again and were permanently stuck with today's frontier models, we're still going to see an incredible technological transformation over the next decade or two like nothing you've ever experienced as general purpose robots are built out and become a fundamental part of everyday life, and all scientific efforts become essentially infinitely parallelizable. That's enabled by _today's_ frontier models becoming widespread and integrated into systems (In this scenario I'm assuming training smaller and narrow AI is still allowed, just not training new frontier general purpose models).
And it's important to understand this is the real reason I'm not gung ho about advancing _despite the risks_. We've already made massive strides in AI that aren't going to be rolled back. And p(immortality | no-superintelligence) is _also_ a decently high number. We don't need to incur a risk of extinction to get things we're going to get anyway
I should've been more clear. I think p(doom|superintelligence) is very high. I think p(superintelligence) is not necessarily that high. So p(doom) is not necessarily that high. But this does mean p(superintelligence & not-doom) is very low. So the upside scenario is very unlikely, and not something we should be gambling with doom to try to achieve.
Of course there's no hard numbers on this, but it's my impression from seeing nearly no one argue that both superintelligence and alignment are in our near future. Few have a vision of that
When you say you don't think p(superintelligence) is high, you're saying you think it's unlikely current techniques scale to superintelligence? Or that you think it's likely we will successfully identify where the danger threshold is and stop scaling before we get there (whether through policy or individual lab efforts)?
Some indeterminate share of optimistic forecasts surely prices in the possibility that this technology will be banned or otherwise seriously curbed by governments.
Inference along the lines "forecasters are not too worried about a huge disaster caused by AI, therefore we shouldn't regulate AI" is imho obviously invalid. Perhaps they are in fact predicting that disaster-preventing regulation is going to be enacted.
Yay! Thank you Richard for finally considering p(doom) seriously!
All of the numbers you use for your final estimate of between 2% and 17% are _marginal_ probabilities _given_ whatever coordinated safety efforts are predicted to the degree they succeed at slowing AI progress.
Please be careful not to treat them as like p(doom | no-safety-effort) which should be a higher number and the thing that drives policy decisions.
Safety effort is likely to make P(doom) less likely so even if we can't accurately calculate p(doom|safety-effort) or p(doom|no-safety-effort), it's clear that a safety effort moves things in the right direction.
Up until now Richard has been against AI regulation of any kind. His reasoning has been all the usual (usually correct) reasons for pushing back against Luddism. I'm glad to see him digging into the AI safety debate and asking whether this one really is different.
The whole discussion is polluted by people who think we're worried about job loss or copyright issues or water use as well as by people who (very stupidly IMO) actually _are_ worried about job loss and copyright issues and water use, and it's nice to see Richard firmly step out of that basin
I usually ask for their model of doom, not just a p(doom). Models reveal methods, can be analyzed, and debated. It’s why I take people who are worried about global warming seriously: They’ve modeled it, considered how variables interact, tested the parts they can, and then calibrated their models. My point isn’t a slam dunk, a model is not necessary or sufficient, but having a model separates serious people from those who just make up probabilities “that seem reasonable.” Blind empiricism is not the best path to knowledge. The next question doomers ask me is, “what would a model look like?” and I say, “That’s your job.”
The problem with outside view forecasting methods is the same problem as the AP arguments: how likely was a thing, which would have prevented us observing it if it occurred?
Given that we're here observing, it clearly didn't happen, but that says nothing about how likely it was.
The biggest point in favor of metaculus/rat forecasters to my mind has been their performance on AI up to this point - being consistently more correct than superforcasters about capabilities and behaviors.
The fundamental problem is that AI is completely and fundamentally unprecedented in human history. Forecasters have nothing to fall back on. That’s not to say no one knows anything, but it’s hard to assign confidence in anyone’s predictions, particularly if you’re relying on track record.
> If the payoff to superintelligence is that we cure most diseases and all live like billionaires, then where exactly your estimate falls within that range could determine which path you think we should take.
The chance of doom can be influenced by our actions, and it's not just a binary choice between stasis and a heedless rush to superintelligence. I think few people are advocating for never developing superintelligence at all. But let's say there is a 2% chance of doom if we rush (let's say we get there in 5 years then), and a 1% chance if we slow down and take 3 times as long (so 15 years). Some people will die of disease etc. in the extra time it takes to superintelligence, but on the plus side there is, in expectation, 1% of almost infinite future value.
Of course, this logic only works if you care about the wellbeing of future people - if you only care about those currently alive, you might say: "well more than 1% of people will die in those extra years". Then it becomes more important if your p(doom) is 2% or 15%. Is that the root of the disagreement?
There's a counterfactual where we don't develop superintelligence but _still_ cure most diseases and all live like billionaires, just get there a little bit slower. If not for that, I'd be way more willing to gamble
I would break down the question into 3 parts:
(Q1) How likely is it that humans will pause the development of strong AI, for maybe a decade or so, to find out how to align it better?
(Q2) if there is no pause, how likely is AI to kill us all?
(Q3) if there is a pause, how likely is AI to kill us all?
My answer to Q2 is at least 10% and possibly as high as 95%.
My answer to Q3 is i don't know but it is surely going to make AI doom significantly less likely.
Therefore I am for a pause.
Obviously the creation of a new life form more intelligent than humans has never happened before, so it is clearly an outside context problem, and very hard to calculate proabilities.
What you need to understand is that the only definite part of these estimates is that the chance is not 0, everything after that from specific estimates to ranges is almost entirely vibes and hyperactive imagination here.
Where this pretty clearly breaks down is that the best Metaculus usées _are_, by any reasonable definition, superforecasters themselves.
So we should take their views at least as much into account as the FRI superforecasters, and then consider if there are any reasons to believe one is more likely correct than the other.
My suspicion is that the Metaculus superforecasters are more likely correct than the FRI ones because we have much less information about the FRI one's level of ongoing engagement with forecasting than we do for the metaculus ones, but this is a very weak point. I don't have a stronger one so maybe a mean is appropriate here!
Stop repeating the current propaganda! The current propaganda states we have to build-out data centers to compete with China in the AI race. What nonsense. To use AI for legitimate purposes gigantic data centers are not required. The data centers are needed to surveil and control through Central Bank Digital currency and social credit scores EVERYONE in real time, all day every day, step by step! There is no need to compete with China on implementing totalitarianism unless your goal is totalitarianism.