I'm not in politics but I am in an industry in which prognostication is huge, and I can tell you this flat-out: clients will pay huge amounts on the latest, cutting-edge models, and react with confusion and disappointment if they deviate even slightly from averages and consensuses. I'm halfway convinced that you could slap a nice skin on a simple average of polls, say it has all the latest machine learning technology, and people would line up for it.
People don't want models or numbers They want someone smarter than themselves to just parrot back what they already believe in order to flatter their priors. The most hilarious thing is to follow the comments to Nate Silver's updates and see how he's a paid Thiel-shill or an establishment media woketard depending on the results of the day.
Exactly my experience as well, even with things so comparatively cut and dry as production planning. Way back in the misty dawn of my career I was really behind on getting the production forecast done due to MRP system issues, and a coworker advised “just use the graphs from last month and change the dates, no one will notice.” Turns out she was correct, and in fact they didn’t even notice when I reused a graph and forgot to change the dates.
Not a good practice, but as a test of “does anyone even care or think about this?” it was instructive.
I think the lack of outliers is less about being shook from Trump’s 16/20 overperformance and more about this election being genuinely close.
In 2016/20 the herd was gathered in Safe D territory, so even a significant deviation from the herd in either direction would likely still show a Dem victory and the poll would remain correct in terms of forecasting the winner, thus limiting the potential for embarrassment.
Publishing a +12 Biden poll when Biden may win by 5? Who cares we’ll still get it right. Publishing a +5 Kamala poll when Trump may win by 2? Hell no we’re not publishing that, if we’re wrong we’ll get crucified.
1) Herding exists, especially towards the end of a race.
2) Silver's model (at least what's being used at the Silver Bulletin Substack; no telling what the Mouse is now doing over at 538) tries to penalize herders, as part of a larger effort to grade pollsters on quality. Higher-quality polls have more impact in his model, versus pollsters that follow the herd or have a partisan taint.
This is distinct from correction for house effects where some pollsters tend to lean one way or the other without the use of shenanigans, and Silver corrects for this without penalty. No pollster throws it straight down the middle and they all tend to lean to one side or the other, but some are better than others.
3) Paradoxically, herding increases the accuracy of individual polls but hurts the accuracy of the average.
And Silver would be the first to remind people that this is a very close race, well within the margin for error, and odds approaching that of a coin toss.
About the ONLY thing that might be a tell is when Silver ran his prediction model (emphasis on PREDICTION model) using only A-rated polls like NYT and Fox News (yes, Fox's poll is considered a quality poll), the PREDICTION model tipped a little more in Trump's favor.
This was to answer a question about the possibility of GOP pollsters (looking at you, Rasmussen) "flooding the zone" with garbage polls.
Another way to read this blogpost is that Hanania comes out strongly in favor of frequentism in interpreting polling results. From a Bayesian perspective, the results of a given poll would also take the prior into account, thus producing herding AKA regression towards the mean.
I think the larger issue is that it is much easier to apply Bayesian Reasoning to a large number of Raw polls that apply Bayesian Reasoning to polls that are a mix of Raw, Bayesian adjusted, poorly Bayesian adjusted, and politically biased. Trying to weigh them properly is an extremely difficult problem that we have to rely on sophisticated models like the one Nate Silver created. Silver’s model would be more accurate and simpler if everyone just made Raw polls and he knew that.
What's with the terrible graph at the beginning? I assume you know that it should look like a smooth curve, and only looks that way because of your low sample size? Why simulate only 30?
Its interesting to consider the potential implications - elections are in part a beauty contest - in that some voters won't turn out if the election is polling as a slam dunk for one side or the other.
In a way this should help increase turnout - therefore benefitting Trump (as he probably does better among the "low-information" voters).
It's possible though that there is a "silent majority" of centrists who dislike Trump's populism and rudeness - and this could help turn them out.
I agree with the thrust of your point but not your argument. The potential reason not to trust the polls isn't related to sampling error so looking at the distribution of sampling errors nor poll distribution isn't really informative.
Polling firms may be overly cautious in their choices about how to weight but they fix that approach relatively early so any single pollster's polls are going to end up with the appropriate distribution of sampling error. And indeed, the graphs for 2020 and 2024 look pretty much like that in 2016 and pretty much like the kind of imperfect approximation of a bell curve that one would except.
But you are likely correct that the firms are overly cautious so are essentially forcing the electorate to look like it did in 2020. Some firms are even weighting by the person's recalled vote in 2020. This means two things, it tends to underweight the incumbent because people often falsely remember voting for the winner (tho one could imagine that being reversed this time) and it fails to adjust to population changes (if more Biden voters move to PA you can't pick that up). And even the firms that aren't explicitly doing that no doubt are trying to be extra sure their weighting doesn't repeat the mistake of 2020.
--
Whether that's a good or bad thing probably depends a bit on what you want a poll to do. It's probably does reduce the chance of being really far off in favor of the democrats but it also means it's not going to be able to inform you as well about changes in the mood, feelings etc etc of the electorate if they are moving.
Perhaps this says something about our culture as a whole, that people are less tolerant of high-risk behaviors. As this goes on, we should expect to see increasing conformism.
"I simply don’t believe that six out of seven swing states will be decided by one point or less."
This wouldn't be so unusual. In 2020, 5 states (GA, AZ, WI, PA, and NC) were decided by 1.3 points or less. In a closer election, which this one is, you would expect the results to be... well, closer. You have a point about artificial conformity among recent polls, but it won't be the closeness of the election result that's fishy, it'll be the closeness of the polls.
I think what we're observing is a kind of prisoner's dilemma for pollsters. If every pollster would cooperate and publish outliers, the data would be more diverse and informative. But if only one publishes outliers and others defect, then the lone outlier makes the average less accurate (while reflecting poorly on that pollster as you pointed out), so individual pollsters are incentivized to release only close results.
“As you can see, very few polls give you a result that is exactly 50/50. Across the 30 polls, the results range from a Republican lead of 3.4 percentage points to a Democratic lead of 4.2. Most come close to the mean.”
I'm not in politics but I am in an industry in which prognostication is huge, and I can tell you this flat-out: clients will pay huge amounts on the latest, cutting-edge models, and react with confusion and disappointment if they deviate even slightly from averages and consensuses. I'm halfway convinced that you could slap a nice skin on a simple average of polls, say it has all the latest machine learning technology, and people would line up for it.
People don't want models or numbers They want someone smarter than themselves to just parrot back what they already believe in order to flatter their priors. The most hilarious thing is to follow the comments to Nate Silver's updates and see how he's a paid Thiel-shill or an establishment media woketard depending on the results of the day.
You just explained the entire business model for management consultants.
^^^ THIS
Exactly my experience as well, even with things so comparatively cut and dry as production planning. Way back in the misty dawn of my career I was really behind on getting the production forecast done due to MRP system issues, and a coworker advised “just use the graphs from last month and change the dates, no one will notice.” Turns out she was correct, and in fact they didn’t even notice when I reused a graph and forgot to change the dates.
Not a good practice, but as a test of “does anyone even care or think about this?” it was instructive.
I think the lack of outliers is less about being shook from Trump’s 16/20 overperformance and more about this election being genuinely close.
In 2016/20 the herd was gathered in Safe D territory, so even a significant deviation from the herd in either direction would likely still show a Dem victory and the poll would remain correct in terms of forecasting the winner, thus limiting the potential for embarrassment.
Publishing a +12 Biden poll when Biden may win by 5? Who cares we’ll still get it right. Publishing a +5 Kamala poll when Trump may win by 2? Hell no we’re not publishing that, if we’re wrong we’ll get crucified.
Good point that I should’ve mentioned.
Nate Silver has a couple of oldie-but-goodie articles on the phenomenon of herding:
https://fivethirtyeight.com/features/heres-proof-some-pollsters-are-putting-a-thumb-on-the-scale/
https://fivethirtyeight.com/methodology/how-our-pollster-ratings-work/
TL; DR:
1) Herding exists, especially towards the end of a race.
2) Silver's model (at least what's being used at the Silver Bulletin Substack; no telling what the Mouse is now doing over at 538) tries to penalize herders, as part of a larger effort to grade pollsters on quality. Higher-quality polls have more impact in his model, versus pollsters that follow the herd or have a partisan taint.
This is distinct from correction for house effects where some pollsters tend to lean one way or the other without the use of shenanigans, and Silver corrects for this without penalty. No pollster throws it straight down the middle and they all tend to lean to one side or the other, but some are better than others.
3) Paradoxically, herding increases the accuracy of individual polls but hurts the accuracy of the average.
And Silver would be the first to remind people that this is a very close race, well within the margin for error, and odds approaching that of a coin toss.
About the ONLY thing that might be a tell is when Silver ran his prediction model (emphasis on PREDICTION model) using only A-rated polls like NYT and Fox News (yes, Fox's poll is considered a quality poll), the PREDICTION model tipped a little more in Trump's favor.
This was to answer a question about the possibility of GOP pollsters (looking at you, Rasmussen) "flooding the zone" with garbage polls.
https://www.natesilver.net/p/are-republican-pollsters-flooding
Another way to read this blogpost is that Hanania comes out strongly in favor of frequentism in interpreting polling results. From a Bayesian perspective, the results of a given poll would also take the prior into account, thus producing herding AKA regression towards the mean.
I think the larger issue is that it is much easier to apply Bayesian Reasoning to a large number of Raw polls that apply Bayesian Reasoning to polls that are a mix of Raw, Bayesian adjusted, poorly Bayesian adjusted, and politically biased. Trying to weigh them properly is an extremely difficult problem that we have to rely on sophisticated models like the one Nate Silver created. Silver’s model would be more accurate and simpler if everyone just made Raw polls and he knew that.
Surely no other explanations for this are possible in a world where people are honest and systems are incorruptible.
What's with the terrible graph at the beginning? I assume you know that it should look like a smooth curve, and only looks that way because of your low sample size? Why simulate only 30?
Or the electorate could just be more decided. If you run back into the 90s, you should see even bigger swings over time.
Its interesting to consider the potential implications - elections are in part a beauty contest - in that some voters won't turn out if the election is polling as a slam dunk for one side or the other.
In a way this should help increase turnout - therefore benefitting Trump (as he probably does better among the "low-information" voters).
It's possible though that there is a "silent majority" of centrists who dislike Trump's populism and rudeness - and this could help turn them out.
I agree with the thrust of your point but not your argument. The potential reason not to trust the polls isn't related to sampling error so looking at the distribution of sampling errors nor poll distribution isn't really informative.
Polling firms may be overly cautious in their choices about how to weight but they fix that approach relatively early so any single pollster's polls are going to end up with the appropriate distribution of sampling error. And indeed, the graphs for 2020 and 2024 look pretty much like that in 2016 and pretty much like the kind of imperfect approximation of a bell curve that one would except.
But you are likely correct that the firms are overly cautious so are essentially forcing the electorate to look like it did in 2020. Some firms are even weighting by the person's recalled vote in 2020. This means two things, it tends to underweight the incumbent because people often falsely remember voting for the winner (tho one could imagine that being reversed this time) and it fails to adjust to population changes (if more Biden voters move to PA you can't pick that up). And even the firms that aren't explicitly doing that no doubt are trying to be extra sure their weighting doesn't repeat the mistake of 2020.
--
Whether that's a good or bad thing probably depends a bit on what you want a poll to do. It's probably does reduce the chance of being really far off in favor of the democrats but it also means it's not going to be able to inform you as well about changes in the mood, feelings etc etc of the electorate if they are moving.
Perhaps this says something about our culture as a whole, that people are less tolerant of high-risk behaviors. As this goes on, we should expect to see increasing conformism.
"I simply don’t believe that six out of seven swing states will be decided by one point or less."
This wouldn't be so unusual. In 2020, 5 states (GA, AZ, WI, PA, and NC) were decided by 1.3 points or less. In a closer election, which this one is, you would expect the results to be... well, closer. You have a point about artificial conformity among recent polls, but it won't be the closeness of the election result that's fishy, it'll be the closeness of the polls.
I think what we're observing is a kind of prisoner's dilemma for pollsters. If every pollster would cooperate and publish outliers, the data would be more diverse and informative. But if only one publishes outliers and others defect, then the lone outlier makes the average less accurate (while reflecting poorly on that pollster as you pointed out), so individual pollsters are incentivized to release only close results.
You asked ChatGPT to simulate exactly this.
Am I confused?
“As you can see, very few polls give you a result that is exactly 50/50. Across the 30 polls, the results range from a Republican lead of 3.4 percentage points to a Democratic lead of 4.2. Most come close to the mean.”
The subsequent paragraph wasn't meant as optional reading.
Hi Errorology, my name is Cynicology. Nice to meet you.