The ‘science’ of polling is broken

Pre-election polling failed to capture other equally profound political shifts around the country. Loudon County, Va., a short commute from Washington, D.C., and home to the country club wing of the government bureaucracy, swung nearly 10 points toward Trump. Miami-Dade — one of the last blue bastions in Florida — swung 20 points. Closer to home, Fall River turned from a reliably Democratic stronghold into one of the many New England towns we can now safely regard as Trump territory.

You’d expect to see this kind of sea change reflected in the polls before the election. But we didn’t.

Overall, polling in battleground states may have been marginally more accurate this year than it was in past election cycles. But the fact that polling this year missed the broader shift to the right reveals a more important truth: Polling as we’ve known it over the last century is irreparably broken. And like so many of our broken things, we don’t yet know how to replace it.

The notion of gauging public opinion on anything — from support for fascism to whether people prefer Fluff over peanut butter — traditionally was based on one important thing: the random sample. It became a staple of political polling in 1936, when George Gallup used the method to contradict the conventional wisdom that FDR was headed for a historic defeat.

“As recently as 1983, if you wanted to know what people thought, you made a thousand calls, conducted interviews with the six hundred who answered the phone, and were literally thirty minutes away from releasing the poll,” says Michael A. Bailey, a Georgetown University public policy professor and author of “Polling at a Crossroads: Rethinking Modern Survey Research.”

The size of the sample alone virtually guaranteed a representative cross-section of America. “It’s amazing,” Bailey says: Surveys back then would be accurate on many levels, “down to including the right number of people with diabetes, even if they don’t know they have diabetes.”

That was then; this is now. Decades of declining trust in institutions and the proliferation of spam across all our communication networks have led to an explosive growth in what pollsters call non-response bias, which simply means that the one person out of a hundred who does respond to an unknown caller on their cellphone is unlikely to be representative of most Americans. “By definition they’re weird, right?” says Bailey. “They’re literally one out of a hundred.”

One paragon of the random sampling method — the highly respected Iowa pollster Ann Selzer — shocked politicos and thrilled Harris supporters by releasing a poll the weekend before the election indicating that Harris led Trump in Iowa by three points. Iowa had not voted for a Democrat for president since Barack Obama in 2012. Last week the state went for Trump by 13 points. Selzer was off by 16 points.

When it takes a hundred calls to reach a single likely voter, random samples are neither random nor samples in any meaningful sense. Most polling firms have reacted to this challenge by using ever more elaborate models to “weight” the data they get from the people who respond. They construct representations of our diverse society by relying on data collected via Doodle polls and other online questionnaires and paid surveys. Modeling uses complex mathematical formulas to estimate one district’s political leanings based on factors such as its previous voting records, the demographic breakdown within that district, and how places with similar demographics have previously voted.

In a polarized country that delegates its most consequential elections to a handful of states, these models were mostly focused on pinpointing the sentiments of the residents of these battlegrounds and ignoring the rest of us. Modeling with weighted samples worked great in 2008 and 2012 (while Gallup, still relying on random sampling, underestimated Barack Obama’s support by 9 percentage points). But modeling has had diminishing returns ever since.

In this cycle most pollsters — chastened by their failures to predict the true level of Trump’s support in the last two cycles — weighted their results based on how respondents remembered previously voting. This led to the seemingly accurate results in the swing states because “these models basically said, ‘This vote will be like the last vote,’ and in some places that was more or less true but with a small tilt toward Trump,” notes Bailey.

But that doesn’t mean polls told us an important and true story about America. That story was playing out, largely undetected, in Bergen County and Fall River and a thousand other locations. If you looked at nearly every poll and read coverage from across the political spectrum, you still had no idea that the Obama coalition of unions, urban professionals, and racial minorities had been rent asunder, a demographic earthquake that will require the party to rethink its basic purpose.

“Right now the modeling, for all its sophistication, has a hard time detecting change,” Bailey says. “Polling is in need of a new paradigm.”

The problem is that detecting change is precisely what we depend on polls to do. When they can’t, the world becomes a dimmer, less comprehensible place.

Jeff Howe, a former contributing editor at Wired magazine, is an associate professor of journalism at Northeastern University. Ian Dartley, a freelance reporter and recent graduate of Northeastern’s master’s in journalism program, contributed research for this article.