I think 2% and 17% chances of human extinction prompt very similar policy responses, specifically trying to lower the chances of it happening by a few orders of magnitude in either case.
I think it's a mistake to go "Well we should risk it because the upside is high". A lot of the non-extinction probability space is in AI just never becoming super human. If AI does become smart enough to cure all diseases and enable wide spread space colonization, then it's much more likely to be capable of killing us too. The more likely outcomes are either we die or not much changes. There aren't many scenarios where we're all helped by a merciful machine god.
I'm against unrestricted AI development and my p(doom | unrestricted AI development) is 80% but I disagree with this take. If my p(doom) was zero, I would be an extreme accelerationist. I actually can't envision how a friendly AI superintelligence _wouldn't_ basically overhaul what it means to be human (in a good way).
At every turn, every bottleneck to human advancement is blocked by intelligence and the application of intellectual effort.
If we never made another advance again and were permanently stuck with today's frontier models, we're still going to see an incredible technological transformation over the next decade or two like nothing you've ever experienced as general purpose robots are built out and become a fundamental part of everyday life, and all scientific efforts become essentially infinitely parallelizable. That's enabled by _today's_ frontier models becoming widespread and integrated into systems (In this scenario I'm assuming training smaller and narrow AI is still allowed, just not training new frontier general purpose models).
And it's important to understand this is the real reason I'm not gung ho about advancing _despite the risks_. We've already made massive strides in AI that aren't going to be rolled back. And p(immortality | no-superintelligence) is _also_ a decently high number. We don't need to incur a risk of extinction to get things we're going to get anyway
I should've been more clear. I think p(doom|superintelligence) is very high. I think p(superintelligence) is not necessarily that high. So p(doom) is not necessarily that high. But this does mean p(superintelligence & not-doom) is very low. So the upside scenario is very unlikely, and not something we should be gambling with doom to try to achieve.
Of course there's no hard numbers on this, but it's my impression from seeing nearly no one argue that both superintelligence and alignment are in our near future. Few have a vision of that
When you say you don't think p(superintelligence) is high, you're saying you think it's unlikely current techniques scale to superintelligence? Or that you think it's likely we will successfully identify where the danger threshold is and stop scaling before we get there (whether through policy or individual lab efforts)?
You don't think a scenario where future AIs turn out to be 100x as smart as GPT and Claude, but just as congenial and obedient to humans, is at all likely? Why not?
Given the current level of niceness and obedience of Claude and ChatGPT, an AI 100x smarter would certainly cause a major catastrophe at minimum. Current models are not well aligned, and regularly cheat in evaluations.
I lived with a roommate who had a couple of dogs: Trux and Otto. They would always fight over "the ball". There were many balls, but Trux only wanted the smallest one. Otto only wanted whichever ball Trux wanted.
Otto was smarter than Trux and regardless of which dog you gave the ball to, Otto would somehow end up with it. So whenever there was a dispute (which was always) I gave the ball to Trux. Hopefully Otto didn't think that meant I liked Trux more or something.
When it was time to eat, Otto would forget about the ball and Trux could get it back. Eventually Trux figured this out and one day asked for food (nose pointed at food bag). I poured some food into Trux's bowl and he made a face like I'd tricked him. He actually wanted me to give _Otto_ food so Trux could get the ball back. Later Otto started putting the ball in his food bowl while he ate.
Trux had to be let out into the backyard to do his business. We didn't have a doggy door. We were all-powerful gods who could open and close doors and Trux lived in a universe where doors opened and closed at our discretion without his say. Sometimes it would rain and Trux hated rain. He thought rain was us playing a mean joke on him. If you can open doors why can't you control rain? I don't want to go out back in the rain. Please stop the rain so I can go out and poop without getting wet?
Was this all an excuse to talk about cute dogs acting silly? Mostly. But the point is that dogs are dumb. Their preferences are simple and have little to do with the larger world. Most of the decisions that impact their lives aren't things they'll ever be able to understand. Often what they want is nonsensical or in conflict or confused or bad. It wouldn't even make sense to _only_ care about a dog's wants once you're at human-level intelligence.
The things we want likely stop being coherent or meaningful to a being sufficiently beyond us, in the same way a dog's preferences are incoherent and pointless to us. There isn't an easy fix for this problem.
I think if we achieve that, someone using them jail breaking them and achieving massive cyber attacks or biological warfare is much too likely.
Or with 100x intelligence, they start recursively improving themselves and achieve 10000x intelligence. And something that smart is very dangerous and hard to control.
Some indeterminate share of optimistic forecasts surely prices in the possibility that this technology will be banned or otherwise seriously curbed by governments.
Inference along the lines "forecasters are not too worried about a huge disaster caused by AI, therefore we shouldn't regulate AI" is imho obviously invalid. Perhaps they are in fact predicting that disaster-preventing regulation is going to be enacted.
I usually ask for their model of doom, not just a p(doom). Models reveal methods, can be analyzed, and debated. It’s why I take people who are worried about global warming seriously: They’ve modeled it, considered how variables interact, tested the parts they can, and then calibrated their models. My point isn’t a slam dunk, a model is not necessary or sufficient, but having a model separates serious people from those who just make up probabilities “that seem reasonable.” Blind empiricism is not the best path to knowledge. The next question doomers ask me is, “what would a model look like?” and I say, “That’s your job.”
I think it's a mistake to assume a doomer model would look anything like other scientific models. The hypothesis is that AIs will eventually be far smarter than humans, and that this will allow them to control or exterminate humans in ways that we cannot even envision. Thus it makes no sense to ask "What sort of technology will future ASIs use to control or kill us?" If we knew that, then by definition it would not require superintelligence. This is an unusual issue, not easy to model.
I get why people find this state of affairs to be frustrating--I find it frustrating. But I cannot think of any logical reason to completely reject the hypothesis. So I remain agnostic.
I also think it's a mistake to assume that experts are drawn to these ideas merely due to what Richard calls mood affiliation. When I read papers by people like Yudkowsky, they seem quite rational. I certainly would not be able to win a debate with him on this issue, so on what basis should I completely reject his concerns?
I think this understates things. Superintelligent AI is impossible to predict for the same reason I can't predict stockfish's moves. But even subhuman level AI is far far harder to predict than climate. For many reasons, chiefly that there isn't precedent for such things and agentic behavior by its nature is harder to predict than the ~climate~
I think it’s unusual too, but I’d like the doomers to try to model it. Otherwise we’re mostly talking in vague terms about something that doesn’t exist and filling in the gaps with probabilities that some people claim are “reasonable.”
Many of their arguments assume the conclusion and works backwards. That’s not the case with a formalized model. I’m not saying a model resolves it, but it does make me very nervous when somebody is trying to convince me of something without showing me their best work.
And scott's point mostly true, though i think the behavior of subhuman AI systems is ALSO far less predictable than the climate. Just because they are goal-directed agents we have zero precedence for.
And I know you complain about this very point, but if something like the AI-2027 forecasts don't count then what would count iye? Can you make a model to explain why things will be fine?
Yay! Thank you Richard for finally considering p(doom) seriously!
All of the numbers you use for your final estimate of between 2% and 17% are _marginal_ probabilities _given_ whatever coordinated safety efforts are predicted to the degree they succeed at slowing AI progress.
Please be careful not to treat them as like p(doom | no-safety-effort) which should be a higher number and the thing that drives policy decisions.
Safety effort is likely to make P(doom) less likely so even if we can't accurately calculate p(doom|safety-effort) or p(doom|no-safety-effort), it's clear that a safety effort moves things in the right direction.
Up until now Richard has been against AI regulation of any kind. His reasoning has been all the usual (usually correct) reasons for pushing back against Luddism. I'm glad to see him digging into the AI safety debate and asking whether this one really is different.
The whole discussion is polluted by people who think we're worried about job loss or copyright issues or water use as well as by people who (very stupidly IMO) actually _are_ worried about job loss and copyright issues and water use, and it's nice to see Richard firmly step out of that basin
One thing that may be worth considering is that a lot of superforecasters underrate AI capabilities progress a lot (check out https://claude.ai/code/artifact/8b0a21d1-bda1-41a4-aa2d-883668ef8e37#no_universal_links), and most people have a bad record of predicting more complex interactions (i.e. with the economy). I think the thing that makes the most sense is to find the person/group that’s been most consistently right and defer to them. I think this is probably Daniel Kokotajlo, but his predictions for non-capabilities things aren’t nearly as good as his capabilities predictions, so it’s hard to say.
Also worth noting that Eli Lifland, Ajeya Cotra, and Peter Wildeford are all superforecasters with relatively higher p(dooms)
The fundamental problem is that AI is completely and fundamentally unprecedented in human history. Forecasters have nothing to fall back on. That’s not to say no one knows anything, but it’s hard to assign confidence in anyone’s predictions, particularly if you’re relying on track record.
>When experts are giving you a range of literally <1% to 100% on a question, it indicates that people are talking past one another, and they lack agreed upon methods to determine how to draw conclusions about an issue.
This is so silly. When experts disagree on whether the R0 for a disease is 0.99 or 1.01, they're not talking past each other or lack shared epidemiological methods but this parameter can have massively different downstream consequences.
"One of the main lessons to come out of Tetlock’s work is that the best generalist forecasters are usually better at telling the future than domain-specific experts, even when it comes to those experts’ areas of study. This is because forecasting itself is its own science that requires a unique way of thinking, and a person having a great deal of knowledge about a field is no guarantee that they have good epistemological habits."
I would expect that many rationalist types who use platforms like Metaculus are well-versed in the science of forecasting, and have read books like *Superforecasting* on how to forecast better. Also doesn't Metaculus overweight estimates from forecasters with a stronger track record?
Usually "nothing ever happens" is a pretty good forecast, but AI has already falsified that one. Bryan Caplan has acknowledged that his AI bet is the only one he's taken which he anticipates losing: https://www.betonit.ai/p/gpt-retakes-my-midterm-and-gets-an
"But they have a unique view on this topic and have had trouble convincing other smart people to take the threat of misalignment anywhere near as seriously as they do."
There are a bunch of open letters like this, and they generally have a bunch of high-profile signatories. Here's another: https://www.pacingthefrontier.com/
The problem with outside view forecasting methods is the same problem as the AP arguments: how likely was a thing, which would have prevented us observing it if it occurred?
Given that we're here observing, it clearly didn't happen, but that says nothing about how likely it was.
The biggest point in favor of metaculus/rat forecasters to my mind has been their performance on AI up to this point - being consistently more correct than superforcasters about capabilities and behaviors.
I roughly agree with most of this article, but the last section is totally insane. For 99.99% of humanity's existence nobody had ever had a flying car. Should we therefore conclude that we are unlikely to get flying cars in the near future?
I normally think highly of your analysis, but this is not a serious approach to forecasting. Of course we can look at the state of technological progress and conclude that the future is going to differ from the past.
Richard does a meta-analysis of various predictions, and finds 17% as the highest probability of human extinction. But we cannot morally equivocate between all forms of extinction. The extinction of the dinosaurs was bad, but since mammals replaced them, it wasn’t that bad. It would have been worse if mammals died out too. A silicon-based intelligence will find a use for hydrocarbon slaves, and life will persist. If you take insect welfare seriously, then there’s a huge difference between “extinct due to asteroid / nuclear holocaust” and “extinct, but replaced by bugs.” So-called “Rationalists” fail to distinguish between the two outcomes, because they are actually not so rational, and are instead hysterical, emotional, autistic babies. Their high openness disguises this, but they are just as vulnerable to moral panics as everyone else.
There is an assumption that an AI who does not care about human suffering will do something that either leads to human extinction and only leaves itself (we are unclear if it experiences pain and pleasure and welfare, and if that would outweigh the genocide) and any lifeforms it creates (unlikely to feel pleasure if the AI does not care about that), or to mass suffering of humans.
An AI which would create a new life that can feel pleasure/pain, and which has fulfilling lives, is to a degree aligned, even if not with humans. This leap seems unlikely, and is the reason it has not been addressed so far - or perhaps people just have not thought of this scenario yet.
Maybe you should bring it up to the community, see their perspective (the 10 comments on your insect people article are not enough), instead of shrilly calling them “autistic babies“ beforehand. That would be much more honest.
I think hydrocarbons are too efficient to be extinguished. Lithium batteries suck. I'm also not a utilitarian or hedonist. I say that risking extinction is worth it.
"Metaculus Pro Forecasters" are not Metaculus's best forecasters, but rather people who have been selected from among Metaculus's best forecasters according to somewhat opaque and subjective criteria. I worked for Metaculus briefly in 2022/23 and they were riven with incompetence and various forms of misconduct and had no rational decision-making processes. They continue to be a website with at most a few thousand core users that somehow costs 3m/year, employs multiple project managers and communications specialists, etc. They are part of the EA blob with all of the incompetence and weird ideology that implies.
All of these discussions somehow assume that AI is an approximation of a human brain, so will act like a human being, complete with all its impulses and emotions. Because AI is not a human brain (clearly since we don’t have anything close to a model for a brain), all of these expert opinions are nonsense in my opinion.
> If the payoff to superintelligence is that we cure most diseases and all live like billionaires, then where exactly your estimate falls within that range could determine which path you think we should take.
The chance of doom can be influenced by our actions, and it's not just a binary choice between stasis and a heedless rush to superintelligence. I think few people are advocating for never developing superintelligence at all. But let's say there is a 2% chance of doom if we rush (let's say we get there in 5 years then), and a 1% chance if we slow down and take 3 times as long (so 15 years). Some people will die of disease etc. in the extra time it takes to superintelligence, but on the plus side there is, in expectation, 1% of almost infinite future value.
Of course, this logic only works if you care about the wellbeing of future people - if you only care about those currently alive, you might say: "well more than 1% of people will die in those extra years". Then it becomes more important if your p(doom) is 2% or 15%. Is that the root of the disagreement?
There's a counterfactual where we don't develop superintelligence but _still_ cure most diseases and all live like billionaires, just get there a little bit slower. If not for that, I'd be way more willing to gamble
I think 2% and 17% chances of human extinction prompt very similar policy responses, specifically trying to lower the chances of it happening by a few orders of magnitude in either case.
I think it's a mistake to go "Well we should risk it because the upside is high". A lot of the non-extinction probability space is in AI just never becoming super human. If AI does become smart enough to cure all diseases and enable wide spread space colonization, then it's much more likely to be capable of killing us too. The more likely outcomes are either we die or not much changes. There aren't many scenarios where we're all helped by a merciful machine god.
I'm against unrestricted AI development and my p(doom | unrestricted AI development) is 80% but I disagree with this take. If my p(doom) was zero, I would be an extreme accelerationist. I actually can't envision how a friendly AI superintelligence _wouldn't_ basically overhaul what it means to be human (in a good way).
At every turn, every bottleneck to human advancement is blocked by intelligence and the application of intellectual effort.
If we never made another advance again and were permanently stuck with today's frontier models, we're still going to see an incredible technological transformation over the next decade or two like nothing you've ever experienced as general purpose robots are built out and become a fundamental part of everyday life, and all scientific efforts become essentially infinitely parallelizable. That's enabled by _today's_ frontier models becoming widespread and integrated into systems (In this scenario I'm assuming training smaller and narrow AI is still allowed, just not training new frontier general purpose models).
And it's important to understand this is the real reason I'm not gung ho about advancing _despite the risks_. We've already made massive strides in AI that aren't going to be rolled back. And p(immortality | no-superintelligence) is _also_ a decently high number. We don't need to incur a risk of extinction to get things we're going to get anyway
I should've been more clear. I think p(doom|superintelligence) is very high. I think p(superintelligence) is not necessarily that high. So p(doom) is not necessarily that high. But this does mean p(superintelligence & not-doom) is very low. So the upside scenario is very unlikely, and not something we should be gambling with doom to try to achieve.
Of course there's no hard numbers on this, but it's my impression from seeing nearly no one argue that both superintelligence and alignment are in our near future. Few have a vision of that
When you say you don't think p(superintelligence) is high, you're saying you think it's unlikely current techniques scale to superintelligence? Or that you think it's likely we will successfully identify where the danger threshold is and stop scaling before we get there (whether through policy or individual lab efforts)?
I think current techniques probably don't scale all the way to super intelligence. But it's hard to know.
You don't think a scenario where future AIs turn out to be 100x as smart as GPT and Claude, but just as congenial and obedient to humans, is at all likely? Why not?
Given the current level of niceness and obedience of Claude and ChatGPT, an AI 100x smarter would certainly cause a major catastrophe at minimum. Current models are not well aligned, and regularly cheat in evaluations.
I lived with a roommate who had a couple of dogs: Trux and Otto. They would always fight over "the ball". There were many balls, but Trux only wanted the smallest one. Otto only wanted whichever ball Trux wanted.
Otto was smarter than Trux and regardless of which dog you gave the ball to, Otto would somehow end up with it. So whenever there was a dispute (which was always) I gave the ball to Trux. Hopefully Otto didn't think that meant I liked Trux more or something.
When it was time to eat, Otto would forget about the ball and Trux could get it back. Eventually Trux figured this out and one day asked for food (nose pointed at food bag). I poured some food into Trux's bowl and he made a face like I'd tricked him. He actually wanted me to give _Otto_ food so Trux could get the ball back. Later Otto started putting the ball in his food bowl while he ate.
Trux had to be let out into the backyard to do his business. We didn't have a doggy door. We were all-powerful gods who could open and close doors and Trux lived in a universe where doors opened and closed at our discretion without his say. Sometimes it would rain and Trux hated rain. He thought rain was us playing a mean joke on him. If you can open doors why can't you control rain? I don't want to go out back in the rain. Please stop the rain so I can go out and poop without getting wet?
Was this all an excuse to talk about cute dogs acting silly? Mostly. But the point is that dogs are dumb. Their preferences are simple and have little to do with the larger world. Most of the decisions that impact their lives aren't things they'll ever be able to understand. Often what they want is nonsensical or in conflict or confused or bad. It wouldn't even make sense to _only_ care about a dog's wants once you're at human-level intelligence.
The things we want likely stop being coherent or meaningful to a being sufficiently beyond us, in the same way a dog's preferences are incoherent and pointless to us. There isn't an easy fix for this problem.
I think if we achieve that, someone using them jail breaking them and achieving massive cyber attacks or biological warfare is much too likely.
Or with 100x intelligence, they start recursively improving themselves and achieve 10000x intelligence. And something that smart is very dangerous and hard to control.
Some indeterminate share of optimistic forecasts surely prices in the possibility that this technology will be banned or otherwise seriously curbed by governments.
Inference along the lines "forecasters are not too worried about a huge disaster caused by AI, therefore we shouldn't regulate AI" is imho obviously invalid. Perhaps they are in fact predicting that disaster-preventing regulation is going to be enacted.
I usually ask for their model of doom, not just a p(doom). Models reveal methods, can be analyzed, and debated. It’s why I take people who are worried about global warming seriously: They’ve modeled it, considered how variables interact, tested the parts they can, and then calibrated their models. My point isn’t a slam dunk, a model is not necessary or sufficient, but having a model separates serious people from those who just make up probabilities “that seem reasonable.” Blind empiricism is not the best path to knowledge. The next question doomers ask me is, “what would a model look like?” and I say, “That’s your job.”
What do you really mean by a model then? Something like AI-2027 feels like a good mainline scenario for how things will go.
Something that looks like a climate model, not Termination Shock.
I think it's a mistake to assume a doomer model would look anything like other scientific models. The hypothesis is that AIs will eventually be far smarter than humans, and that this will allow them to control or exterminate humans in ways that we cannot even envision. Thus it makes no sense to ask "What sort of technology will future ASIs use to control or kill us?" If we knew that, then by definition it would not require superintelligence. This is an unusual issue, not easy to model.
I get why people find this state of affairs to be frustrating--I find it frustrating. But I cannot think of any logical reason to completely reject the hypothesis. So I remain agnostic.
I also think it's a mistake to assume that experts are drawn to these ideas merely due to what Richard calls mood affiliation. When I read papers by people like Yudkowsky, they seem quite rational. I certainly would not be able to win a debate with him on this issue, so on what basis should I completely reject his concerns?
I think this understates things. Superintelligent AI is impossible to predict for the same reason I can't predict stockfish's moves. But even subhuman level AI is far far harder to predict than climate. For many reasons, chiefly that there isn't precedent for such things and agentic behavior by its nature is harder to predict than the ~climate~
I think it’s unusual too, but I’d like the doomers to try to model it. Otherwise we’re mostly talking in vague terms about something that doesn’t exist and filling in the gaps with probabilities that some people claim are “reasonable.”
Many of their arguments assume the conclusion and works backwards. That’s not the case with a formalized model. I’m not saying a model resolves it, but it does make me very nervous when somebody is trying to convince me of something without showing me their best work.
Ok, then show me what a formalized doomer model would look like.
The supplemental material I'm talking abut with forecasts and justifications and whatnot
https://ai-2027.com/research
And scott's point mostly true, though i think the behavior of subhuman AI systems is ALSO far less predictable than the climate. Just because they are goal-directed agents we have zero precedence for.
And I know you complain about this very point, but if something like the AI-2027 forecasts don't count then what would count iye? Can you make a model to explain why things will be fine?
Yay! Thank you Richard for finally considering p(doom) seriously!
All of the numbers you use for your final estimate of between 2% and 17% are _marginal_ probabilities _given_ whatever coordinated safety efforts are predicted to the degree they succeed at slowing AI progress.
Please be careful not to treat them as like p(doom | no-safety-effort) which should be a higher number and the thing that drives policy decisions.
Safety effort is likely to make P(doom) less likely so even if we can't accurately calculate p(doom|safety-effort) or p(doom|no-safety-effort), it's clear that a safety effort moves things in the right direction.
Up until now Richard has been against AI regulation of any kind. His reasoning has been all the usual (usually correct) reasons for pushing back against Luddism. I'm glad to see him digging into the AI safety debate and asking whether this one really is different.
The whole discussion is polluted by people who think we're worried about job loss or copyright issues or water use as well as by people who (very stupidly IMO) actually _are_ worried about job loss and copyright issues and water use, and it's nice to see Richard firmly step out of that basin
Too much obsession with the word 'extinction', as with climate change doomerism.
Untrammelled AI (and CC) could generate plenty of miserable outcomes without causing extinction. Isn't that bad enough?
One thing that may be worth considering is that a lot of superforecasters underrate AI capabilities progress a lot (check out https://claude.ai/code/artifact/8b0a21d1-bda1-41a4-aa2d-883668ef8e37#no_universal_links), and most people have a bad record of predicting more complex interactions (i.e. with the economy). I think the thing that makes the most sense is to find the person/group that’s been most consistently right and defer to them. I think this is probably Daniel Kokotajlo, but his predictions for non-capabilities things aren’t nearly as good as his capabilities predictions, so it’s hard to say.
Also worth noting that Eli Lifland, Ajeya Cotra, and Peter Wildeford are all superforecasters with relatively higher p(dooms)
The fundamental problem is that AI is completely and fundamentally unprecedented in human history. Forecasters have nothing to fall back on. That’s not to say no one knows anything, but it’s hard to assign confidence in anyone’s predictions, particularly if you’re relying on track record.
>When experts are giving you a range of literally <1% to 100% on a question, it indicates that people are talking past one another, and they lack agreed upon methods to determine how to draw conclusions about an issue.
This is so silly. When experts disagree on whether the R0 for a disease is 0.99 or 1.01, they're not talking past each other or lack shared epidemiological methods but this parameter can have massively different downstream consequences.
"One of the main lessons to come out of Tetlock’s work is that the best generalist forecasters are usually better at telling the future than domain-specific experts, even when it comes to those experts’ areas of study. This is because forecasting itself is its own science that requires a unique way of thinking, and a person having a great deal of knowledge about a field is no guarantee that they have good epistemological habits."
I would expect that many rationalist types who use platforms like Metaculus are well-versed in the science of forecasting, and have read books like *Superforecasting* on how to forecast better. Also doesn't Metaculus overweight estimates from forecasters with a stronger track record?
Usually "nothing ever happens" is a pretty good forecast, but AI has already falsified that one. Bryan Caplan has acknowledged that his AI bet is the only one he's taken which he anticipates losing: https://www.betonit.ai/p/gpt-retakes-my-midterm-and-gets-an
"But they have a unique view on this topic and have had trouble convincing other smart people to take the threat of misalignment anywhere near as seriously as they do."
See the signatories here, which include two of the world's most cited researchers and a bunch of other big shots: https://aistatement.com/work/statement-on-ai-extinction-risk
There are a bunch of open letters like this, and they generally have a bunch of high-profile signatories. Here's another: https://www.pacingthefrontier.com/
The problem with outside view forecasting methods is the same problem as the AP arguments: how likely was a thing, which would have prevented us observing it if it occurred?
Given that we're here observing, it clearly didn't happen, but that says nothing about how likely it was.
The biggest point in favor of metaculus/rat forecasters to my mind has been their performance on AI up to this point - being consistently more correct than superforcasters about capabilities and behaviors.
I roughly agree with most of this article, but the last section is totally insane. For 99.99% of humanity's existence nobody had ever had a flying car. Should we therefore conclude that we are unlikely to get flying cars in the near future?
I normally think highly of your analysis, but this is not a serious approach to forecasting. Of course we can look at the state of technological progress and conclude that the future is going to differ from the past.
Richard does a meta-analysis of various predictions, and finds 17% as the highest probability of human extinction. But we cannot morally equivocate between all forms of extinction. The extinction of the dinosaurs was bad, but since mammals replaced them, it wasn’t that bad. It would have been worse if mammals died out too. A silicon-based intelligence will find a use for hydrocarbon slaves, and life will persist. If you take insect welfare seriously, then there’s a huge difference between “extinct due to asteroid / nuclear holocaust” and “extinct, but replaced by bugs.” So-called “Rationalists” fail to distinguish between the two outcomes, because they are actually not so rational, and are instead hysterical, emotional, autistic babies. Their high openness disguises this, but they are just as vulnerable to moral panics as everyone else.
There is an assumption that an AI who does not care about human suffering will do something that either leads to human extinction and only leaves itself (we are unclear if it experiences pain and pleasure and welfare, and if that would outweigh the genocide) and any lifeforms it creates (unlikely to feel pleasure if the AI does not care about that), or to mass suffering of humans.
An AI which would create a new life that can feel pleasure/pain, and which has fulfilling lives, is to a degree aligned, even if not with humans. This leap seems unlikely, and is the reason it has not been addressed so far - or perhaps people just have not thought of this scenario yet.
Maybe you should bring it up to the community, see their perspective (the 10 comments on your insect people article are not enough), instead of shrilly calling them “autistic babies“ beforehand. That would be much more honest.
I think hydrocarbons are too efficient to be extinguished. Lithium batteries suck. I'm also not a utilitarian or hedonist. I say that risking extinction is worth it.
Can we not rely on future warning shots to help clarify the actual risk of AI catastrophe?
Hi Richard,
"Metaculus Pro Forecasters" are not Metaculus's best forecasters, but rather people who have been selected from among Metaculus's best forecasters according to somewhat opaque and subjective criteria. I worked for Metaculus briefly in 2022/23 and they were riven with incompetence and various forms of misconduct and had no rational decision-making processes. They continue to be a website with at most a few thousand core users that somehow costs 3m/year, employs multiple project managers and communications specialists, etc. They are part of the EA blob with all of the incompetence and weird ideology that implies.
All of these discussions somehow assume that AI is an approximation of a human brain, so will act like a human being, complete with all its impulses and emotions. Because AI is not a human brain (clearly since we don’t have anything close to a model for a brain), all of these expert opinions are nonsense in my opinion.
> If the payoff to superintelligence is that we cure most diseases and all live like billionaires, then where exactly your estimate falls within that range could determine which path you think we should take.
The chance of doom can be influenced by our actions, and it's not just a binary choice between stasis and a heedless rush to superintelligence. I think few people are advocating for never developing superintelligence at all. But let's say there is a 2% chance of doom if we rush (let's say we get there in 5 years then), and a 1% chance if we slow down and take 3 times as long (so 15 years). Some people will die of disease etc. in the extra time it takes to superintelligence, but on the plus side there is, in expectation, 1% of almost infinite future value.
Of course, this logic only works if you care about the wellbeing of future people - if you only care about those currently alive, you might say: "well more than 1% of people will die in those extra years". Then it becomes more important if your p(doom) is 2% or 15%. Is that the root of the disagreement?
There's a counterfactual where we don't develop superintelligence but _still_ cure most diseases and all live like billionaires, just get there a little bit slower. If not for that, I'd be way more willing to gamble