Which Experts Should You Trust on AI Doom?

A lot depends on the forecasters you ask

Scott Sumner has some wise words on trying to assess p(doom) as an outsider. In his article, I came across this result of the 2022 chart below on existential risk. Experts here give extinction by 2100 somewhere between a 0.1% and 100% chance, and the guesses are roughly equally distributed across that range, at least if you use a log scale.

A more recent survey from December 2024 presents this figure.

At the 25th, 50th, and 75th percentiles of concern, the probabilities of extinction or extreme disempowerment are 1%, 10%, and 25%. Yet a quarter of researchers estimate the odds to be higher, all the way up to 100%.

What is a non-expert supposed to do with these numbers? Generally, when there is an expert consensus, you can probably trust it. But here, the data is telling us that the field can’t agree on anything.

When experts are giving you a range of literally <1% to 100% on a question, it indicates that people are talking past one another, and they lack agreed upon methods to determine how to draw conclusions about an issue.

With a distribution like this, I don’t think you can just take the mean or median and consider that a reasonable guess. There’s a difference between a 10% probability of doom in which the vast majority of specialists are between 5% and 15% – in which case I’d be inclined to trust the experts – and one where the estimates are all over the place.

Instead of looking at experts in AI, we might ask what the superforecasters think. A report from the Forecasting Research Institute (FRI) explains how, between June and October 2022, the organization brought together 80 specialists on long-run existential risk and 89 forecasters with a history of accurate predictions.

What’s interesting here is that domain-level experts were more worried across the board. At the median, experts thought that there was a 20% chance of a catastrophic risk by 2100, which meant at least 10% of humans dying within any 5-year period, compared to 9% for superforecasters. Experts at the median gave a 6% chance of human extinction by 2100, which is defined as fewer than 5,000 people alive. For superforecasters, the number was 1%.

The chart below shows the distribution of answers for superforecasters, AI experts, experts in other domains, and a public survey on the question of AI catastrophic risk.

We see the superforecasters not only having a lower estimate, but clustering much closer together.

This report was from 2022, which was a very long time ago in terms of AI capabilities. But things haven’t changed that much among superforecasters. As of mid-2026, superforecasters thought that there was a 2.4% chance of an AI catastrophe by 2100, as defined above, compared to 5% for experts. Respondents were asked about their estimates depending on whether AI progress was fast or slow by 2030, with the results shown below. Note again how the experts have such wide confidence intervals, compared to the superforecasters.

Estimates from mid-2026

Note that experts here are not all specialists in AI. They are described as “44 computer scientists, 37 industry professionals, 45 economists, and 68 research staff at policy think tanks.” If you look at the median estimates for catastrophic AI risk by 2100 for each group, you get the following:

Computer scientists — 8%

Policy experts — 7%

AI industry professionals — 3%

Economists — 3%

The estimates of doom among the best forecasters are much higher at Metaculus, though it depends on how the question is framed. Metaculus uses a weighted median based on each participant’s most recent update. Among pro forecasters, as of this writing we find a 36% chance of a catastrophe from any cause in which humanity suffers an at least 10% population decline within a 5-year span by 2100. Conditional on such a catastrophe happening, they give a 67% chance it results from AI. And if we get an AI catastrophe, the pro forecasters say there is a 72% chance that we see an at least 95% population decline.

Multiplying those three probabilities together, you get 0.36 x 0.67 x 0.72 = 0.17 as a probability that AI reduces the number of humans by at least 95%.

At the same time, however, if you ask whether humans will go extinct by 2100, pro forecasters only give a 7.5% probability.

Obviously, it is impossible for the odds of human extinction due to AI to be higher than the odds of human extinction for any reason. There could be a logical reason for the disparity, since all humans going extinct is different from an at least 95% population decline. So perhaps pro forecasters think there is an approximately 10% chance that we lose 95% of humans but that there are a few survivors. I doubt this is the case, and it’s more likely that framing effects explain the results.

It’s worth noting here that the Metaculus community as a whole is much less pessimistic than the pro forecasters. All users of the site give a 3% chance of extinction from any cause, and an implied 4.3% chance of an at least 95% population decline as a result of AI. That puts them close to the FRI forecasters.

When it comes to the most successful users of Metaculus, however, no matter which estimate we use, they think doom is much more likely than those affiliated with FRI. The table below shows the most recent estimates for each group discussed so far.

The difference between 2.4% and 7%-17% is quite large. And the gap is not driven by recent news events, as the pro forecasters at Metaculus have consistently thought that there was a higher risk of extinction or catastrophe going back to 2022, though the gap has grown over time. Note also that the table likely underestimates the current difference between FRI forecasters and AI experts, given that the survey of the latter was in late 2024.

Why is there so much disagreement between these various groups? The best explanation is probably sociological. Metaculus grew out of the rationalist community, where people are obsessed with the idea of doom. Meanwhile, the Forecasting Research Institute is rooted in Phil Tetlock’s academic work on expertise and making predictions, which has been a project of academic psychologists and economists from the beginning.

We’ve already seen that China is less fearful of doom, likely due to that country not having as much of a rationalist subculture, and not being as afraid of the future or as pessimistic about technology. Now, we can see that, in the United States, there are large differences even among people who are considered high-quality forecasters, based on their connections to the rationalist community.

Again, maybe the rationalists are right. But they have a unique view on this topic and have had trouble convincing other smart people to take the threat of misalignment anywhere near as seriously as they do. They’re having more success in the court of public opinion, but this is not surprising, as voters are very dumb and anxious, and always put ridiculously high estimates on nightmare scenarios. AI experts are also pretty close to the rationalists at Metaculus in their estimates, but I think that the overlap between the categories of “AI experts who answer surveys on AI” and “rationalists” is quite large.

So who should you trust? One of the main lessons to come out of Tetlock’s work is that the best generalist forecasters are usually better at telling the future than domain-specific experts, even when it comes to those experts’ areas of study. This is because forecasting itself is its own science that requires a unique way of thinking, and a person having a great deal of knowledge about a field is no guarantee that they have good epistemological habits. If you’re going to ask, for example, whether American democracy is falling apart, an expert might start by making a superficial comparison between the modern US and the Weimar Republic. Someone with a forecasting mindset would be more likely to begin with a question like “how often do advanced democracies slide into dictatorships?” The answer is not very often.

Based on Tetlock’s work, and without a good reason to think otherwise, I would tend to trust the superforecasters more than the AI experts. The fact that, as mentioned before, AI expert estimates are all over the place, seems to indicate that they are even less trustworthy than experts in other domains, at least on this question. I doubt that if you asked economists about the odds of a recession in the next few years, you would get that large of a range. Then again, the Metaculus crowd complicates things, since their selected superforecasters are much more pessimistic than the FRI crowd.

I have no strong feelings here, and will therefore say that a good guess for the probability of doom is probably somewhere between 2% and 17%. On the one hand, you have both the Chinese and Tetlock-affiliated superforecasters not really treating misalignment as an existential problem. On the other, I tend to trust rationalists and think highly of their judgment on most issues, since they tend to be for things like genetic engineering and assisted suicide. So who knows?

Here, someone might say that it doesn’t matter where we land, and that a risk of either 2% or 17% is too high. That would be true if AI didn’t have a chance to be a transformational technology. If the payoff to superintelligence is that we cure most diseases and all live like billionaires, then where exactly your estimate falls within that range could determine which path you think we should take.

I would go with base rates, and this is probably how most non-expert superforecasters are thinking. A base rate analysis says:

1. Humans have never gone extinct.

2. Since perhaps the Black Death in the fourteenth century, our species has never even come close to suffering a disaster on the scale of 10% of humanity dying over five years, or a 10% population decline, which are the two definitions used for catastrophic risk in the forecasts cited above. And the Black Death was hundreds of years before the Industrial Revolution started us on the path of sustained economic growth, since which large die-offs as a share of the human population have become even rarer.

3. New market-based technologies are almost always good.

That’s what we know. Most other analysis seems to be speculation based on mood affiliation, no matter how close individuals are to AI technology.

This post is sponsored by Mechanize. AI alignment is perhaps the most important issue of our time, and if you’re interested in being part of that effort, Mechanize is hiring software engineers at a base salary of $300K-$400K/year, plus substantial bonuses. They also have openings for other positions, including Recruiter ($200K). Here is their website, and you can apply here. If you apply, make sure to use the links provided here so I can receive credit, and name me as a reference in the process.