Pluralistic: When prophecy fails, election polling edition

In Canto 20 of Inferno, Dante confronts a pit where the sinners have had their heads twisted around backwards; they trudge, naked and weeping, through puddles of cooling tears. Virgil informs him that these are the fortunetellers, who tried to look forwards in life and now must look backwards forever.

In a completely unrelated subject, how about those election pollsters, huh?

Writing for The American Prospect, historian Rick Perlstein takes a hard look at characteristic failure modes of election polling and ponders their meaning:

https://prospect.org/politics/2024-09-25-polling-imperilment/

Apart from the pre-election polling chaos we're living through today, Perlstein's main inspiration is W Joseph Campbell 2024 University of California Press book, Lost in a Gallup: Polling Failure in US Presidential Elections:

https://www.ucpress.edu/books/lost-in-a-gallup/paper

In Campbell's telling, US election polling follows a century-old pattern: pollsters discover a new technique that works spookily well..for a while. While the new polling technique works, the pollster is hailed a supernaturally insightful fortune-teller.

In 1932, the Raleigh News and Observer was so impressed with polling by The Literary Digest that they proposed replacing elections with Digest's poll. The Digest's innovation was sending out 20,000,000 postcards advertising subscriptions and asking about presidential preferences. This worked perfectly for three elections - 1924, 1928, and 1932. But in 1936, the Digest blew it, calling the election for Alf Landon over FDR.

The Digest was dethroned, and new soothsayers was appointed: George Gallup, Elmo Roper and Archibald Crossler, who replaced the Digest's high-volume polling with a new kind of poll, one that sought out a representative slice of the population (as Perlstein says, this seems "so obvious in retrospect, you wonder how nobody thought of it before").

Representative polling worked so well that, three elections later, the pollsters declared that they could predict the election so well from early on that there was no reason to keep polling voters. They'd just declare the winner after the early polls were in and take the rest of the election off.

That was in 1948 - you know, 1948, the "Dewey Defeats Truman" election?

https://en.wikipedia.org/wiki/Dewey_Defeats_Truman

If this sounds familiar, perhaps you - like Perlstein - are reminded of the 2016 election, where Fivethirtyeight and Nate Silver called the election for Hillary Clinton, and we took them at their word because they'd developed a new, incredibly accurate polling technique that had aced the previous two elections.

Silver's innovation? Aggregating state polls, weighting them by accuracy, and then producing a kind of meta-poll that combined their conclusions.

When Silver's prophecy failed in 2016, he offered the same excuse that Gallup gave in 1948: when voters are truly undecided, you can't predict how they'll vote, because they don't know how they'll vote.

Which, you know, okay, sure, that's right. But if you know that the election can't be called, if you know that undecided voters are feeding noise into the system whenever you poll them, then why report the polls at all? If all the polling fluctuation is undecided voters flopping around, not making up their mind, then the fact that candidate X is up 5 points with undecided means nothing.

As the finance industry disclaimer has it, "past performance is no guarantee of future results." But, as Perlstein says, "past performance is all a pollster has to go on." When Nate Silver weights his model in favor of a given poll, it's based on that poll's historical accuracy, not its future accuracy, because its future accuracy can't be determined until it's in the past. Like Dante's fortune-tellers, pollsters have to look backwards even as they march forwards.

Of course, it doesn't help that in some cases, Silver was just bad at assessing polls for accuracy, like when he put polls from the far-right "shock pollster" Trafalgar Group into the highly reliable bucket. Since 2016, Trafalgar has specialized in releasing garbage polls that announce that MAGA weirdos are way ahead, and because they always say that, they were far more accurate than the Clinton-predicting competition in 2016 when they proclaimed that Trump had it in the bag. For Silver, this warranted an "A-" on reliability, and that is partially to blame for how bad Silver's 2020 predictions were, when Republicans got pasted, but Trafalgar continued to predict a Democratic wipeout. Silver's methodology has a huge flaw: because Trafalgar's prediction history began in 2016, that single data-point made them look pretty darned reliable, even though their method was to just keep saying the same thing, over and over:

https://www.ettingermentum.news/p/the-art-of-losing-a-fivethirtyeight

Pollsters who get lucky with a temporarily reliable methodology inevitably get cocky and start cutting corners. After all, polling is expensive, so discontinuing the polls once you think you have an answer is a way to increase the enterprise's profitability. But, of course, pollsters can only make money so long as they're somewhat reliable, which leads to a whole subindustry of excuse-making when this cost-cutting bites them in the butt. In 1948, George Gallup blamed his failures on the audience, who failed to grasp the "difference between forecasting an election and picking the winner of a horse race." In 2016, Silver declared that he'd been right because he'd given Trump at 28.6% chance of winning.

This isn't an entirely worthless excuse. If you predict that Clinton's victory is 71.4% in the bag, you are saying that Trump might win. But pollsters want to eat their cake and have it, too: when they're right, they trumpet their predictive accuracy, without any of the caveats they are so insistent upon when they blow it:

There's always some excuse when it comes to the polls: in 1952, George Gallup called the election a tossup, but it went for Eisenhower in a landslide. He took out a full-page NYT ad, trumpeting that he was right, actually, because he wasn't accounting for undecided voters.

Polling is ultimately a form of empiricism-washing. The pollster may be counting up poll responses, but that doesn't make the prediction any less qualitative. Sure, the pollster counts responses, but who they ask, and what they do with those responses, is purely subjective. They're making guesses (or wishes) about which people are likely to vote, and what it means when someone tells you they're undecided. This is at least as much an ideological project as it is a scientific one:

https://prospect.org/blogs-and-newsletters/tap/2024-09-23-polling-whiplash/

But for all that polling is ideological, it's a very thin ideology. When it comes to serious political deliberation, questions like "who is likely to vote" and "what does 'undecided' mean" are a lot less important than, "what are the candidates promising to do?" and "what are the candidates likely to do?"

But - as Perlstein writes - the only kind of election journ*lism that is consistently, adequately funded is poll coverage. As a 1949 critic put it, this isn't the "pulse of democracy," it's "its baby talk."

This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.