Two unmentioned possibilities for leftward bias of polls: 1) many are push-polls designed to produce that result and discourage Republican voters; 2) media outlets live in subscription silos and can't afford to offend the people who pay their salaries.
I think #2 is largely the issue, along with various advocacy groups either generating wildly biased polling or torturing poll results until they confess to what the group wants to promote. I can see no other reason why news organizations continue to use polling outfits that produce garbage other than it fits the stories 'reporters' want to write and the subscribers want to read.
If polling was still independent organizations doing pure horse race 'who would you vote for today' questions I suspect the accuracy would be a lot better.
Finally, the root issue is the one that Richard mentioned almost in passing. The clear indication is that people who respond to polls have fundamentally different views than those who do not respond. This is a social problem that no mathematical wizardry or business model modifications are going to fix.
> Finally, the root issue is the one that Richard mentioned almost in passing. The clear indication is that people who respond to polls have fundamentally different views than those who do not respond. This is a social problem that no mathematical wizardry or business model modifications are going to fix.
In principle, all that means is that the correct amount of adjustment to the poll results is dependent on the response rate. It still allows for polling to produce useful, accurate results.
In practice, I think it's worse than that; opinions will shift among the nonresponding populace too, but that shift will be difficult to see via polling.
I never answer polls. I especially would not do so today with the demonization of all Republicans by the current administration and the media, as well as the FBI targeting of conservatives.
Within the last five years I think the general expectation even among friends wanting to communicate is to text instead of call. Why bother picking up calls from strangers, unless you're looking for a solution to your auto warranty expiration problem? It's impressive that most of these pollsters are even in the ballpark given how difficult it is to reach voters, and extraordinary that Trafalgar has adjusted so well.
Until polls can reliably show the depth of candidate commitment -- something that the voters themselves may not be self-aware of! (See: Oz's rise in the polls.) -- they're pretty useless until about a week before the election. Even then, in a close race how October-surprise proof is that number?
Does any pollster have continuous contact with a demographically balanced set of (likely) voters, imitative of the Nielsen TV rating system? I'd think a model where people are paid for their time in an otherwise increasingly unreachable world would solve a lot of these issues. Pollsters could uncover a lot more interesting qualitative and quantitative insights and be better able to predict a range of outcomes like 538 but based on more first hand data.
> If betting markets and polls are about evenly good in theory, we should want betting markets to do the work for us because the costs of running them are low. Meanwhile, pollsters need to keep an army of political scientists employed to figure out how they can correctly gauge public opinion.
This assumes that the prediction markets are independent of the polls. If the markets are primarily using poll results, getting rid of polls will lower the usefulness of the markets.
I take polls and they stop asking me political questions the closer it gets to election time. I feel like they want the media to say Democrats are always winning to psychologically manipulate people into believing that's how everyone thinks. This is the same concept that happens when you ask people the percentage of Black or Transgender Americans and the numbers are so far off the mark. Hammering minority opinions and issues makes people believe there are more of these people than is real. The same thing happens in these polls, imo.
Conundrum; skewed polls are cover for fraud - “see, we told you the Dem would win.” Skewed polls can cause the opponent to misallocate resources. Which situation is true for your district?
We need trusted online exit polling to steer canvassing efforts. Difficult.
But if it turns out that the republicans that answer polls are unrepresentative in that they are likely to vote for Biden or other Dems then you simply ask the people polled who they voted for in 2020 and use that to correct your model.
I mean polling might get harder but the idea that you wouldn't get any useful information from it seems crazy. Indeed, for any good Bayesian there will always be some model that best reflects their all things considered judgement based on past years which can be updated by polling (so worse the problem is the more role for Silver and less role for unmodeled polls).
All of which doesn't even touch on the fact that probably one of the most important uses of polling isn't to see who will win or exactly what issues have more than 50% support but to see changes in support over time and to compare support in different groups other than just democrats and conservatives.
Pollsters already tried to do this in 2020 by asking who they voted for in 2016, and it didn't work. People have a bad memory, or lie, about who they voted for in long passed elections
Another useful thing we get from polls: they expose social desirability bias, which often has relevance for governing. People may be afraid to say, for example, that they want more assertive policing, but as a practical matter they do want more assertive policing. Or another example: people may feel socially compelled to signal that they favor more restrictive abortion policies when in fact they prefer less restrictive policies. Wise politicians need to be aware of and understand these discrepancies.
For the GOP abortion is like flying a helicopter under an overpass. But if I were a GOP politician, I would say something like "I'm prolife and don't believe in murder of the unborn, which is why I support an abortion ban (with exceptions) after 15 weeks." That would provide cover for people who don't mind early abortion but don't want to be perceived as pro-choice (even with bias, support for abortion plummets after the first trimester). Would it be enough to get through a primary with a full abolitionist? Maybe, depends on how many shy pro-choicers there is. Aside: 15 weeks is, interestingly, Desantis' current position...
It seems correct to say that election polls will always tell us *something*. Asking my mother how she feels about the current President on a scale from 1-10 tells me *something*, and any poll of multiple people, conducted with honesty and consistency, should be a better indicator than that.
Trees don't grow to the sky. The Dem bias in polls has an upward bound, especially if we're actively trying to identify it, so smart pollsters will find that upward bound at some point. If, after all of our adjustments, we can only conclude that polls favor Dems between 1 and 5 points, that's still useful information.
What it seems we're really saying is that the days are over when a very good analyst could somewhat reliably get a near-perfect prediction on the Electoral College. But polling worse than you did last time still means you'll *probably* do worse than last time.
> The Dem bias in polls has an upward bound, especially if we're actively trying to identify it, so smart pollsters will find that upward bound at some point. If, after all of our adjustments, we can only conclude that polls favor Dems between 1 and 5 points, that's still useful information.
That is true, but we could easily find that after making all the adjustments we can, we end up stating that polls favor the Democrats by between -15 and +12 points, which isn't useful information.
Sure, but we're nowhere close to there. And numbers that high probably require things like a large percentage of the people you poll to secretly coordinate to screw up the poll: "This year, the random number generator says all Republicans whose first name contains the letter 'o' should claim to be voting Democrat."
Otherwise, we can largely surmise the political leanings of the unpollable section of the electorate and do all we can to understand who we have managed to poll and how correlated they are with the unpollable.
The main risk you run is that the unpollable section of the electorate swings in a huge way for a particular election and that this is essentially undetectable in the pollable section of the electorate. Though realistically that's probably only going to happen in a Presidential election, not a midterm. We could especially imagine polls being a terrible predictor if there were another Ross Perot type who pulled from both parties (and whose supporters were mostly unpollable), though I don't think in today's environment it's possible to imagine such a candidate -- Americans are a lot farther apart than in 1992.
"If betting markets and polls are about evenly good in theory, we should want betting markets to do the work for us because the costs of running them are low".
This seems wrong to me, because the betting market forecasters are partly relying on the polls data. Take away the polls, and surely the betting markets will deteriorate.
Mr. Hanania asserts "On the night of November 8 we’ll know who was right and who was wrong." How? One presumes the pollsters are attempting to predict how masses of people will actually vote. Why? You can't predict whether 16 ounces of hamburger will weigh a pound until you know a) the acuracy of the scale, and b) how heavily the butcher will "adjust" that scakle with his thumb. If you predict a 1 percent Republican victory, and the Democrats win by 1 percent after surveillance videos show the vote-counters (all poll WATCHERS having been excluded) pulling suitcases full of ballots of indeterminate origin out from under their tables and running them through the scanners again and again till they get the totals they seek, on what basis are you going to conclude the pollster who predicted a 1 percent Republican victory was "wrong by 2 points"? Pollsters could save a lot of trouble by skipping the calls to likely voters, instead simply calling whoever's going to be in charge of the counting rooms in the major swing-state cities, and asking, "After you stop the counting around 2 a.m., and then finish counting the next day, what's your goal as to the percentage by which you plan to arrange for Democrats to win your state?" Providing you could ever get a straight answer, I submit you'd be AMAZED by its accuracy.
The real question is why do we even need polls. They are of no value to me. I make my voting decisions on the issues, not the polls. Most of the polls are so undervalued that the margin of error mokes them impossible to interpret.
On issue polling, Goodhart's law seems like a factor: parties have an incentive to structure their rhetoric, using motte and bailey tactics, so as to make their positions seem as noncontroversially right as possible, so that they can produce polls indicating broad public support, which is politically useful in it's own right.* The constant competition to out-manipulate poll respondents will definitely frustrate their reliability at assessing underlying beliefs.
*possible weakness of this argument is I'm assuming it is actually politically useful to be able to produce a poll saying '73% of people support my policy.'
It is useful in the sense that it helps you convince politicians that they will benefit from taking your position. Or at least not lose.
Parties also have an incentive to produce actual unbiased polls that reflect people's real opinions on issues so that they can make good decisions on which policies to push in order to maximize political success.
All polling obviously became crap after the Great Awokening , circa 2014. After that media affiliated pollsters just aren't trustworthy, and no one trusts them. As a result, it would be impossible for them to do their job accurately, even if they wanted to.
BW, How do candidates do their internal polls? Those are likely to be far more accurate because the candidates have real skin in the game regarding resource allocation.
And re: Silver's last remark I think you are missing an important distinction. He says it would take 2 more cycles to convince him the polls were hopelessly biased but I doubt that's what he means by unfixable. I suspect he means something like: impossible to build a model which serves as an unbiased estimator of the outcome.
The elephant in the room remains though, if polls can't be trusted at all will we all just live in darkness every election until the votes are actually counted?
Two unmentioned possibilities for leftward bias of polls: 1) many are push-polls designed to produce that result and discourage Republican voters; 2) media outlets live in subscription silos and can't afford to offend the people who pay their salaries.
I think #2 is largely the issue, along with various advocacy groups either generating wildly biased polling or torturing poll results until they confess to what the group wants to promote. I can see no other reason why news organizations continue to use polling outfits that produce garbage other than it fits the stories 'reporters' want to write and the subscribers want to read.
If polling was still independent organizations doing pure horse race 'who would you vote for today' questions I suspect the accuracy would be a lot better.
Finally, the root issue is the one that Richard mentioned almost in passing. The clear indication is that people who respond to polls have fundamentally different views than those who do not respond. This is a social problem that no mathematical wizardry or business model modifications are going to fix.
"torturing poll results until they confess to what the group wants to promote" is a nice turn of phrase
> Finally, the root issue is the one that Richard mentioned almost in passing. The clear indication is that people who respond to polls have fundamentally different views than those who do not respond. This is a social problem that no mathematical wizardry or business model modifications are going to fix.
In principle, all that means is that the correct amount of adjustment to the poll results is dependent on the response rate. It still allows for polling to produce useful, accurate results.
In practice, I think it's worse than that; opinions will shift among the nonresponding populace too, but that shift will be difficult to see via polling.
I always lie to the pollsters, so do several other people I know.
It's like those race check boxes on forms.
Why? Do you just want to see the world burn?
I never answer polls. I especially would not do so today with the demonization of all Republicans by the current administration and the media, as well as the FBI targeting of conservatives.
Within the last five years I think the general expectation even among friends wanting to communicate is to text instead of call. Why bother picking up calls from strangers, unless you're looking for a solution to your auto warranty expiration problem? It's impressive that most of these pollsters are even in the ballpark given how difficult it is to reach voters, and extraordinary that Trafalgar has adjusted so well.
Until polls can reliably show the depth of candidate commitment -- something that the voters themselves may not be self-aware of! (See: Oz's rise in the polls.) -- they're pretty useless until about a week before the election. Even then, in a close race how October-surprise proof is that number?
Does any pollster have continuous contact with a demographically balanced set of (likely) voters, imitative of the Nielsen TV rating system? I'd think a model where people are paid for their time in an otherwise increasingly unreachable world would solve a lot of these issues. Pollsters could uncover a lot more interesting qualitative and quantitative insights and be better able to predict a range of outcomes like 538 but based on more first hand data.
> If betting markets and polls are about evenly good in theory, we should want betting markets to do the work for us because the costs of running them are low. Meanwhile, pollsters need to keep an army of political scientists employed to figure out how they can correctly gauge public opinion.
This assumes that the prediction markets are independent of the polls. If the markets are primarily using poll results, getting rid of polls will lower the usefulness of the markets.
I take polls and they stop asking me political questions the closer it gets to election time. I feel like they want the media to say Democrats are always winning to psychologically manipulate people into believing that's how everyone thinks. This is the same concept that happens when you ask people the percentage of Black or Transgender Americans and the numbers are so far off the mark. Hammering minority opinions and issues makes people believe there are more of these people than is real. The same thing happens in these polls, imo.
Conundrum; skewed polls are cover for fraud - “see, we told you the Dem would win.” Skewed polls can cause the opponent to misallocate resources. Which situation is true for your district?
We need trusted online exit polling to steer canvassing efforts. Difficult.
> We need trusted online exit polling to steer canvassing efforts.
Do we? Do we need well-targeted canvassing efforts?
But if it turns out that the republicans that answer polls are unrepresentative in that they are likely to vote for Biden or other Dems then you simply ask the people polled who they voted for in 2020 and use that to correct your model.
I mean polling might get harder but the idea that you wouldn't get any useful information from it seems crazy. Indeed, for any good Bayesian there will always be some model that best reflects their all things considered judgement based on past years which can be updated by polling (so worse the problem is the more role for Silver and less role for unmodeled polls).
All of which doesn't even touch on the fact that probably one of the most important uses of polling isn't to see who will win or exactly what issues have more than 50% support but to see changes in support over time and to compare support in different groups other than just democrats and conservatives.
Pollsters already tried to do this in 2020 by asking who they voted for in 2016, and it didn't work. People have a bad memory, or lie, about who they voted for in long passed elections
Another useful thing we get from polls: they expose social desirability bias, which often has relevance for governing. People may be afraid to say, for example, that they want more assertive policing, but as a practical matter they do want more assertive policing. Or another example: people may feel socially compelled to signal that they favor more restrictive abortion policies when in fact they prefer less restrictive policies. Wise politicians need to be aware of and understand these discrepancies.
For the GOP abortion is like flying a helicopter under an overpass. But if I were a GOP politician, I would say something like "I'm prolife and don't believe in murder of the unborn, which is why I support an abortion ban (with exceptions) after 15 weeks." That would provide cover for people who don't mind early abortion but don't want to be perceived as pro-choice (even with bias, support for abortion plummets after the first trimester). Would it be enough to get through a primary with a full abolitionist? Maybe, depends on how many shy pro-choicers there is. Aside: 15 weeks is, interestingly, Desantis' current position...
It seems correct to say that election polls will always tell us *something*. Asking my mother how she feels about the current President on a scale from 1-10 tells me *something*, and any poll of multiple people, conducted with honesty and consistency, should be a better indicator than that.
Trees don't grow to the sky. The Dem bias in polls has an upward bound, especially if we're actively trying to identify it, so smart pollsters will find that upward bound at some point. If, after all of our adjustments, we can only conclude that polls favor Dems between 1 and 5 points, that's still useful information.
What it seems we're really saying is that the days are over when a very good analyst could somewhat reliably get a near-perfect prediction on the Electoral College. But polling worse than you did last time still means you'll *probably* do worse than last time.
> The Dem bias in polls has an upward bound, especially if we're actively trying to identify it, so smart pollsters will find that upward bound at some point. If, after all of our adjustments, we can only conclude that polls favor Dems between 1 and 5 points, that's still useful information.
That is true, but we could easily find that after making all the adjustments we can, we end up stating that polls favor the Democrats by between -15 and +12 points, which isn't useful information.
Sure, but we're nowhere close to there. And numbers that high probably require things like a large percentage of the people you poll to secretly coordinate to screw up the poll: "This year, the random number generator says all Republicans whose first name contains the letter 'o' should claim to be voting Democrat."
Otherwise, we can largely surmise the political leanings of the unpollable section of the electorate and do all we can to understand who we have managed to poll and how correlated they are with the unpollable.
The main risk you run is that the unpollable section of the electorate swings in a huge way for a particular election and that this is essentially undetectable in the pollable section of the electorate. Though realistically that's probably only going to happen in a Presidential election, not a midterm. We could especially imagine polls being a terrible predictor if there were another Ross Perot type who pulled from both parties (and whose supporters were mostly unpollable), though I don't think in today's environment it's possible to imagine such a candidate -- Americans are a lot farther apart than in 1992.
"If betting markets and polls are about evenly good in theory, we should want betting markets to do the work for us because the costs of running them are low".
This seems wrong to me, because the betting market forecasters are partly relying on the polls data. Take away the polls, and surely the betting markets will deteriorate.
Mr. Hanania asserts "On the night of November 8 we’ll know who was right and who was wrong." How? One presumes the pollsters are attempting to predict how masses of people will actually vote. Why? You can't predict whether 16 ounces of hamburger will weigh a pound until you know a) the acuracy of the scale, and b) how heavily the butcher will "adjust" that scakle with his thumb. If you predict a 1 percent Republican victory, and the Democrats win by 1 percent after surveillance videos show the vote-counters (all poll WATCHERS having been excluded) pulling suitcases full of ballots of indeterminate origin out from under their tables and running them through the scanners again and again till they get the totals they seek, on what basis are you going to conclude the pollster who predicted a 1 percent Republican victory was "wrong by 2 points"? Pollsters could save a lot of trouble by skipping the calls to likely voters, instead simply calling whoever's going to be in charge of the counting rooms in the major swing-state cities, and asking, "After you stop the counting around 2 a.m., and then finish counting the next day, what's your goal as to the percentage by which you plan to arrange for Democrats to win your state?" Providing you could ever get a straight answer, I submit you'd be AMAZED by its accuracy.
The real question is why do we even need polls. They are of no value to me. I make my voting decisions on the issues, not the polls. Most of the polls are so undervalued that the margin of error mokes them impossible to interpret.
On issue polling, Goodhart's law seems like a factor: parties have an incentive to structure their rhetoric, using motte and bailey tactics, so as to make their positions seem as noncontroversially right as possible, so that they can produce polls indicating broad public support, which is politically useful in it's own right.* The constant competition to out-manipulate poll respondents will definitely frustrate their reliability at assessing underlying beliefs.
*possible weakness of this argument is I'm assuming it is actually politically useful to be able to produce a poll saying '73% of people support my policy.'
It is useful in the sense that it helps you convince politicians that they will benefit from taking your position. Or at least not lose.
Parties also have an incentive to produce actual unbiased polls that reflect people's real opinions on issues so that they can make good decisions on which policies to push in order to maximize political success.
All polling obviously became crap after the Great Awokening , circa 2014. After that media affiliated pollsters just aren't trustworthy, and no one trusts them. As a result, it would be impossible for them to do their job accurately, even if they wanted to.
BW, How do candidates do their internal polls? Those are likely to be far more accurate because the candidates have real skin in the game regarding resource allocation.
Pollsters and drug dealers have much in common except that polling is legal and a much safer profession.
And re: Silver's last remark I think you are missing an important distinction. He says it would take 2 more cycles to convince him the polls were hopelessly biased but I doubt that's what he means by unfixable. I suspect he means something like: impossible to build a model which serves as an unbiased estimator of the outcome.
The elephant in the room remains though, if polls can't be trusted at all will we all just live in darkness every election until the votes are actually counted?