Fatima and the Sample Size Compensation (SSC) Fallacy

Don't trust groups of peasants, even ones that are very large

Surprisingly, there’s been a great deal of miracle talk among rationalist and rationalist-adjacent Substackers. See Ethan Muse in favor of the 1917 Our Lady of Fatima miracle and Evan Harkness-Murphy against. I didn’t think that the evidence for the miracle was very strong from those two pieces, but then Scott Alexander wrote a deep dive that concluded we cannot dismiss the idea that something supernatural happened, which he only updated slightly in response to the commentary to his original piece.

A key part of Scott’s argument is that he’s impressed that so many people claimed to see the miracle, which involved the sun or something that looked like the sun dancing around or spinning. He writes:

Another common response calls the Sun Miracle a “mass hallucination”. Can 70,000 people really hallucinate the same thing?…

None of these are “mass hallucinations” in the sense where the sorts of visual hallucinations typical of certain mentally ill people occur en masse in a crowd of thousands with >50% prevalence - that is, the type of mass hallucination that would be required to explain Fatima. As far as I know, there are no confirmed cases of this ever happening.

“Confirmation” of such a mass delusion is difficult, since it would involve proving a negative. Regardless, in the follow-up, Scott discusses Iranians seeing Khomeini’s face in the moon, which he doesn’t take seriously, so that appears to qualify, though it’s hard to see why Fatima should be considered more credible, other than the fact that there are rationalist-adjacent Substackers who are Christians but no Shia Muslims.

The bigger problem is that the entire chain of reasoning here rests on a fallacy. Imagine that you’re a pollster trying to figure out who is going to win the 2024 election. You sample a hundred people, and a majority say they are going to vote for Kamala Harris. A MAGA comes along and argues that Democrats are more likely to talk to pollsters, so we should dismiss your survey. You then go back and conduct a new poll, but this time you get a sample of 10,000, and Kamala leads by the same amount. Problem solved?

Of course not! A larger n is a plus if the sampling method is unbiased. But if the sample is biased, talking to more people does not fix the issue. I asked ChatGPT whether this fallacy had a widely accepted name, but it couldn't find one and suggested I coin my own. I kind of like its recommendation of the “Sample Size Compensation Fallacy,” (SSC!) so I’ll go with that.¹ Maybe after this we might call it the Fatima Fallacy, given the central importance of the alleged miracle to modern Catholicism.

Assume that among people predisposed to see a miracle, 70% will hallucinate something they perceive as supernatural happening if they stare at the sun in the midst of a crowd of like-minded individuals. If a crowd is made up of 10 people, 7 say they saw something, and 3 don’t. You would presumably expect the same ratio with a crowd of 1,000 or 100,000. I don’t think Scott would be writing about Fatima if there were only 10 people there at the time, while an alleged 70,000 seems more credible. But this is no different than solving a polling bias by getting a larger sample. It harkens back to the old joke about the business that loses money on each transaction but tries to make up for it in volume.

In fact, the larger crowd size might drive up the portion of people suffering a delusion. If you only have 10 people, the 3 who don’t see anything might feel confident in contradicting the rest. But what if you’re in a giant crowd where 70% are aggressively convinced they witnessed a miracle? Maybe in a crowd of 10, the rate of delusion is 70%, but in a crowd of 70,000 it gets up to 90%, which would approximately match Scott’s original calculations that 5 out of 60 testimonials were negative. It’s also theoretically possible that errors cancel out, but I see little reason to think that would happen given the predisposition of most of the crowd.

We know that groups of religious believers can look at the sun when literally nothing is happening and become convinced that they’re in the presence of a miracle. Scott himself provides video examples of this, one of which I am posting below for reference. If that’s the case, there’s nothing about Fatima having a larger crowd that makes this more impressive. A video of a few dozen people deluding themselves into believing they see a miracle should be taken as strong evidence that the same is possible among a crowd of tens of thousands.

But weren’t there also a lot of skeptics there? Scott writes, “The witnesses included journalists, atheists, prominent scientists, and people who freely admitted that they had only attended in order to laugh at everyone else when nothing happened.” The only confirmed skeptic I know of from reading the previous articles is Avelino de Almeida. Harkness-Murphy writes the following about him:

Avelino was certainly a journalist — a top-shelf journalist, even, judging from his article about Fatima and everything I’ve been able to find out about him. His freethinking credentials are difficult to doubt too: he’s described as the editor of the paper O Século, a periodical founded by someone whose own anticlerical credentials cannot be doubted. But this was the extent of the independent corroboration that I can make for his supposed freethinking agenda. For instance: I have searched unsuccessfully for the apparently Voltairian piece of impious wit he is said to have written before the event took place to mock those who came actually expecting a miracle. It is often cited to but rarely provided, it seems, in material about Fatima. It is not in the Critical Documents as far as I can tell — only Avelino’s now-famous articles from after the miracle appear to be included. Just how skeptical was he, exactly? Even if we can’t pin an exact answer to this question down, I do think Avelino is an excellent witness — and certainly the strongest individual witness on Muse’s side.

Ok, fair enough, we have one guy who was probably a skeptic and witnessed the miracle. Do we have any others? I emailed Scott asking for examples, and he gave me the following:

- A nobleman, Luis Antonio Vieira de Magalhaes e Vasconcelos, who stated that “I was absolutely convinced that I would see nothing”.

- An engineer, Mario Godinho, who said that “In spite of what had happened to my mother [who had attended an earlier apparition], I was disappointed and did not believe in the apparitions. So I sat inside my car. Then all at once I noticed that everybody looked at the sky. Natural curiosity attracted my attention, and I got out of the car and looked at the sky, too.”

- Antonio de Olivera, who said that “At first I did not believe.”

- Carlos Mendes, a lawyer who tried to meet the child-seers earlier. They refused to talk to him, and he said he “left Fatima absolutely disillusioned and convinced by the attitude of the children that there was much here to question. If they had just seen and spoken to the Mother of Heaven, I could not understand why they cried and asked to be left alone. I told several people of my disillusion and publicly declared that I would not go to Fatima on October 13th. My brother, Augusto, a medical doctor who was home from the war in France on leave, intended to go. But I persuaded him not to. Another of my brothers, Candido, had promised to take several persons to Fatima and insisted that I accompany him. I repeatedly refused. Candido and I slept in the same room. At 4 A.M. I heard him getting up to leave for Fatima. I jumped out of the bed, and always protesting that I would not go, I went along.”

- Joaquim Vicente, a quarryman (described by John Haffert): “[He] had come out of curiosity, and was standing with his umbrella over himself and his wife as protection against the rain. In his deposition of February 16, 1960, he tells us: “It was raining. Suddenly the clouds opened and the sun seemed as though through a window. It began spinning, coming so low that I looked at my watch and believed it was not right as the sun was so low. The sun was spinning and there were clouds of different colors like those of the rainbow. Near me a man fell on his knees and exclaimed, ‘My God, forgive me.’ And my wife asked me: ‘So now you don’t believe?’ I answered: ‘If the Church approves, I will believe.’”

All of these people report seeing the miracle.

I also twice messaged Ethan, the guy who wrote the pro-miracle piece, and he never responded. From Scott’s list, note the interesting fact that not a single one of these people is on record as being skeptical before the event happened. The closest thing to an exception is Carlos Mendes, but by his own testimony he seems to have only become a skeptic after being bitterly disillusioned while still having a deep emotional investment in what was going on.

Imagine you hear that there was a miracle at a church service in some remote Kentucky town. Richard Dawkins, Sam Harris, and Christopher Hitchens were all there and report seeing it. That would be quite strong evidence that something strange happened, because each of these individuals has a documented record of atheism. Alternatively, imagine that among the crowd of Kentucky hill people a few retrospectively say that they were skeptics but then God spoke to them. Obviously, that’s completely different. Reading Scott’s original quote, I was expecting testimony of a handful of Sam Harris types, but instead all we’ve got, with one exception, is witnesses looking back and telling us how they felt beforehand.

People often tell stories from their lives in ways that make them sound more interesting. Saying I was always a believer is a much less compelling narrative than having been skeptical at first and then becoming convinced. Maybe some of these people were 50/50 on whether they expected to see anything, but got carried away by the energy of the crowd, and then adjusted the story in their heads by telling themselves that they had been much more skeptical than they actually were. Also, I haven’t looked into exactly when all of the testimonies Scott cites were given, but one is mentioned as having been recounted in 1960, 43 years after the event. That is a lot of time to misremember what happened during what for many of these people might have been the most emotionally charged moment of their lives.

It’s like how you will occasionally hear Trump supporters say that they used to be Democrats, until something like Covid or wokeness happened and made them change their politics. Often, you look into their history, and it turns out that they fit the exact profile of the type of Democratic-leaner who you would expect to go for Trump. How many so-called Fatima skeptics in the crowd were in a similar position? Since there seems to be only one individual who was arguably documented as a skeptic beforehand, we have no way of knowing. Perhaps there were others, but since Scott put a lot of hours into researching Fatima and didn’t find any, and neither has Evan apparently, I am going to assume they don’t exist until presented evidence otherwise.

One reason I don’t put much stock in the testimony of early twentieth century Portuguese peasants is that my mom’s side of the family are poor rural people from the third world. They often report on supernatural events in their lives. From a very early age, I never took any of this seriously, with first-hand experience leading me to conclude that they were very bad at thinking and educated Westerners were much more in touch with reality. My Arab relatives would see their deceased family members in dreams and witness miracles large and small in their personal lives, particularly during moments of personal turmoil or tragedy. They would lie constantly in order to avoid social awkwardness, and discounted things said to them by fellow Arabs on those grounds, while considering the words of white Americans to have more credibility. If there was something you needed to believe for tribal reasons, you were shunned by the rest of the community if you dissented. I haven’t done research into the culture of early twentieth century Portuguese peasants, but I wouldn’t be surprised if there were similarities. The fact that so many people showed up expecting a miracle, based on nothing more than the words of a ten-year old girl who claimed to speak with an imaginary being and her cousins, is itself pretty discrediting for that community.

None of the articles I’ve read on Fatima have taken seriously the later social pressure that one could imagine would’ve been applied to get others to claim that something supernatural happened on the day of the supposed miracle. Some have pointed out that the Portuguese regime was anticlerical at the time and there was a secular intelligentsia. That is presented as indicating that it was less likely that a delusional belief in a miracle could have taken off in such an environment, but it can just as easily be argued that the politicized nature of overt religiosity contributed to the community coalescing around a story that could serve as a rallying cry against a hated regime and put their little village in the middle of nowhere at the center of the cosmos. I’m no historian of Portuguese politics, but I don’t think anyone else who has written about this is either, and it’s annoying to see people cite the existence of an anticlerical elite as clearly making claims of a miracle more credible. After all, when Khomeini was seen on the moon, the Shah was still in power.

So we have two layers of social pressure at work here, consisting of people being in a crowd and getting caught up in the moment, and then the issue of how they remember the event when it takes on religious and political importance.

I think people like Scott and Bentham’s Bulldog who place a lot of faith in Portuguese peasants come from families and communities that have been living in a post-Enlightenment world for generations, thereby underestimating the irrationality of those of us who come from more backwards cultures. Even modern religious people believe all kinds of crazy things and get taken in by mass delusions, as again Scott proved through the videos of fake miracles he posted. According to one source I found, about two-thirds of Portuguese children were illiterate in 1920, and it was probably higher among adults, and even higher still in Fatima. Me using my relatives as a comparison group might be underselling how untrustworthy Portuguese peasants of the time were, since everyone I knew could read. Even a relatively educated person in early twentieth century Portugal grew up in a culture that was completely drenched in conformity and superstition. It’s easy to see how educated Westerners can be so used to interacting with individuals with a rational, scientific worldview that they forget how precious of an accomplishment it was to build communities where they can forget what most of the rest of humanity has been like throughout history.

1. UPDATE: Upon publishing this article, I learned that an academic paper has used the phrase Big Data Fallacy to refer to people interpreting larger samples as more indicative of causation. I think that is similar to this but not the exact same thing.