Scott Sumner has some wise words on trying to assess p(doom) as an outsider. In his article, I came across this result of the 2022 chart below on existential risk. Experts here give extinction by 2100 somewhere between a 0.1% and 100% chance, and the guesses are roughly equally distributed across that range, at least if you use a log scale.
A more recent survey from December 2024 presents this figure.
At the 25th, 50th, and 75th percentiles of concern, the probabilities of extinction or extreme disempowerment are 1%, 10%, and 25%. Yet a quarter of researchers estimate the odds to be higher, all the way up to 100%.
What is a non-expert supposed to do with these numbers? Generally, when there is an expert consensus, you can probably trust it. But here, the data is telling us that the field can’t agree on anything.
This is not a paywall. The rest of this article is available to free subscribers of the newsletter. Just leave your email below to continue reading.
This post is sponsored by Mechanize. AI alignment is perhaps the most important issue of our time, and if you’re interested in being part of that effort, Mechanize is hiring software engineers at a base salary of $300K-$400K/year, plus substantial bonuses. They also have openings for other positions, including Recruiter ($200K). Here is their website, and you can apply here. If you apply, make sure to use the links provided here so I can receive credit, and name me as a reference in the process.
I think 2% and 17% chances of human extinction prompt very similar policy responses, specifically trying to lower the chances of it happening by a few orders of magnitude in either case.
I think it's a mistake to go "Well we should risk it because the upside is high". A lot of the non-extinction probability space is in AI just never becoming super human. If AI does become smart enough to cure all diseases and enable wide spread space colonization, then it's much more likely to be capable of killing us too. The more likely outcomes are either we die or not much changes. There aren't many scenarios where we're all helped by a merciful machine god.
Yay! Thank you Richard for finally considering p(doom) seriously!
All of the numbers you use for your final estimate of between 2% and 17% are _marginal_ probabilities _given_ whatever coordinated safety efforts are predicted to the degree they succeed at slowing AI progress.
Please be careful not to treat them as like p(doom | no-safety-effort) which should be a higher number and the thing that drives policy decisions.