Scott Sumner has some wise words on trying to assess p(doom) as an outsider. In his article, I came across this result of the 2022 chart below on existential risk. Experts here give extinction by 2100 somewhere between a 0.1% and 100% chance, and the guesses are roughly equally distributed across that range, at least if you use a log scale.
A more recent survey from December 2024 presents this figure.
At the 25th, 50th, and 75th percentiles of concern, the probabilities of extinction or extreme disempowerment are 1%, 10%, and 25%. Yet a quarter of researchers estimate the odds to be higher, all the way up to 100%.
What is a non-expert supposed to do with these numbers? Generally, when there is an expert consensus, you can probably trust it. But here, the data is telling us that the field can’t agree on anything.
This is not a paywall. The rest of this article is available to free subscribers of the newsletter. Just leave your email below to continue reading.
This post is sponsored by Mechanize. AI alignment is perhaps the most important issue of our time, and if you’re interested in being part of that effort, Mechanize is hiring software engineers at a base salary of $300K-$400K/year, plus substantial bonuses. They also have openings for other positions, including Recruiter ($200K). Here is their website, and you can apply here. If you apply, make sure to use the links provided here so I can receive credit, and name me as a reference in the process.
I think 2% and 17% chances of human extinction prompt very similar policy responses, specifically trying to lower the chances of it happening by a few orders of magnitude in either case.
I think it's a mistake to go "Well we should risk it because the upside is high". A lot of the non-extinction probability space is in AI just never becoming super human. If AI does become smart enough to cure all diseases and enable wide spread space colonization, then it's much more likely to be capable of killing us too. The more likely outcomes are either we die or not much changes. There aren't many scenarios where we're all helped by a merciful machine god.
Some indeterminate share of optimistic forecasts surely prices in the possibility that this technology will be banned or otherwise seriously curbed by governments.
Inference along the lines "forecasters are not too worried about a huge disaster caused by AI, therefore we shouldn't regulate AI" is imho obviously invalid. Perhaps they are in fact predicting that disaster-preventing regulation is going to be enacted.