The Best and the Brightest

(This is first of a series of posts about the potential pitfalls of quantitative metrics in politics. The second post of the series is here!)

In November 2016, I was sat in the cafeteria of a graduate student dorm building, feeling the collective disbelief growing among the collection of nerds, every laptop rapidly cycling the usual tabs: Five Thirty Eight¹, the New York Times needle, @Redistrict on Twitter. I had that fall just started on a pathway that seemed to lead towards academia, but everything I was thinking about suddenly felt far less important than responding to the political earthquake. The “elevator pitch” for a particular subset of the class of operatives that came of age in 2016 tends to rhyme: we had anticipated a trajectory for our lives² but the election ripped away a substrate we didn’t realize was unsteady. Something broke; I knew I wanted to fix it. But it was also liberating (exonerating?) to know there was nothing I could have done in 2016.

Fast forward eight years later and I was now something of an operative myself, or at least on the cusp of that being a reasonable identifier – a young pretender. I led a data science team that buttressed a significant fraction of the Democratic fundraising juggernaut. I helped build sophisticated models that felt important in bringing close races in the previous midterms over the finish line. It was important work that contributed to redemptive election cycles³.

And then. 

It will be hard to forget the feeling of being in one of the worst possible places on election night: a basement in Georgetown with spotty service surrounded by colleagues who had given their professional lives – and more – for this. Once again, the spectre of the imminent defeat had a physical weight as it stifled the bar.

This time felt different⁴: I had no plausible deniability any more. All of the efforts of our best and our brightest simply weren’t enough.

So I am here to add a voice to the Discourse⁵ of the Reckoning⁶. I have no illusions about my perspective being the magic bullet that will fix it all. Think of it more as, I am inviting you to share in the stream-of-consciousness that keeps me awake at night. And I will strive to stay in my lane – with the hope that the lane of [mathematician-turned-data scientist whose Roman Empire is “postmortems of the Vietnam War”⁷] is not too crowded.

What I mean is: I’m not a political strategist; I’ve only done some volunteer data work on one campaign, having spent my career at agencies. But I’m certified good-at-math, and I have spent my entire professional career thinking about caveats and complications⁸.

This series of essays I’m planning will try to address the fundamental role of data science and statistical inference in modern politics⁹. And hopefully I can first persuade you that this is not an easy question¹⁰.

So where to begin?

Despite having this comic metaphorically pinned on my forehead¹¹, my entry into political work was that of the caricatured physyicst, and certainly the above complaint is common to pretty much any domain expert that saw an influx of data scientists foisted upon them as collaborators. In fact, this is my second round in such a role, having first worked with biologists as a bioinformatician.

In my admittedly short career as a political data scientist, the fundamental tension in my work has been between the political instincts and wisdom of seasoned operatives on one side and measurable, quantitative metrics on the other¹², and expertise often won. To be clear, it’s not that expertise is always correct, especially in these constantly unprecedented times; well-designed experiments done by many of the data scientists and statisticians in our space have done important work in reversing incorrect orthodoxies. But instinct was valuable. Why?

It’s important to know that we don’t have metrics for everything that is important to a political campaign, and we need to be careful to not over-index our decision-making on just that which we can measure. My goal in this series is to persuade you of this.

We’ll first discuss the McNamara fallacy¹³, named after Robert McNamara, the infamous Secretary of Defense at the start of the Vietnam War. It refers to making decisions entirely dependent on quantitative metrics, even when those quantitative metrics miss important aspects of the problem. It’s worth quoting Daniel Yankelovich, who coined the term, in full here¹⁴:

The part of the “definition”, sometimes overshadowed, that always hits me hardest is the bias toward measure only what is most easily measured. It’s easy to see how the incentives governing data work, not just in politics but especially so given the urgency at which campaigns move, lead to this outcome.

Then, we’ll lay out, both with some mathematical detail (drawing from econometrics and systems theory especially) and political examples, some mechanisms by which we might be particularly susceptible to the fallacy¹⁵.

• Nonlinearity: In some sense, this is the substrate underlying all of the complexities of making decisions with data¹⁶. Data scientists rely on models, which use a limited number of features to describe the world they care about. When the world reacts unpredictably to slight changes to or misspecifications of those features, you could be in trouble. Think about, for example, phase transitions: If you were modeling the volume of a gas at different temperatures, you’d conclude that it decreases linearly with temperature¹⁷ … until it turned into a liquid, and your model went out the window.
• Dynamic endogeneity: We’ll go into the technical definition of endogeneity (really quite a broad class of concerns) and especially its prevalence in time-series modeling¹⁸ in the actual essay, but broadly, the type I mean here is that sometimes your explanatory variables today are influenced by your dependent variables yesterday, which screws up your ability to determine causality. Insidiously, this can even affect RCTs – especially their validity over longer time horizons.
• Temporality: Similar to the above, the relationships among the variables you’re using to explain or predict some outcome could themselves be dynamic. A new entrant to the campaign, a particularly viral speech, or a natural disaster could scramble the explanatory power of your models. Today’s RCT, if it was held again next week, might have completely different results¹⁹.
• Network effects: Political identity exists in the matrix of social connections²⁰. There are numerous ways that network effects can confound methods that focus only on individuals, but to name a couple here: the standard errors in your models could be underestimated if there are large covariances between the responses of sets of individuals; and the networks within which people make their political decisions and declarations can itself be endogenous – you could have friends on both sides of your RCT who talk to each other afterwards, for example. 

I’m not an expert in these ideas, but one of the goals of Blue Noise Labs is to gain that expertise. I’d like to invite any and all of the operatives and data scientists out there to help me understand our role in winning elections again – especially if you are an expert²¹.

Footnotes

One of the formative reading experiences of my childhood (and having reread in grad school, it holds up) was The Bartimaeus Trilogy. The chapters in the djinn character’s point of view were rich with hilarious footnotes (to illustrate the characters’ multi-dimensional concsiousness), and I’ve found it a fun way to add some mix of lighthearted asides along with more tangential information/commentary.

1. RIP ↩︎
2. With varying levels of conviction, to be sure ↩︎
3. I worked at BallotReady for the 2020 cycle and MissionWired for 2022 and 2024 ↩︎
4. In so many ways that many others have catalogued, but I mean here, “so different for me” ↩︎
5. ™ ↩︎
6. ™ ↩︎
7. Why is this relevant? The sharp-eyed reader will note the title. Bear with me. ↩︎
8. I’m sure my colleagues will tell you this was endearing ↩︎
9. One thing worth noting: I don’t really have much to say about AI yet, because I myself am only just coming to terms with how it’s being used. My focus here is on “classical” data science. Maybe it will go back to being called statistics… ↩︎
10. And for better or worse, I plan to subject you to my own writing here ↩︎
11. My own version of Hofstadter’s Law ↩︎
12. No prizes for guessing which side I was usually on ↩︎
13. It’s not an exaggeration to say that The Best and the Brightest by David Halberstam, a chronicle of the institutional failures that led to disaster of Vietnam and the inspiration for this entire series, made me question everything I ever though I believed about data-driven decision-making. ↩︎
14. What’s particularly fascinating is that this was evidently coined in reference to McNamara’s time at Ford from before the war, even though the philosophy clearly stayed with him ↩︎
15. One thing I want to emphasize is that these challenges can afflict even the gold standard of experiments, the randomized controlled trial. The point I want to make is that what political operatives have to deal with that scientists don’t is time. Scientific laws are meant to hold across time; political laws might not last the week ↩︎
16. “Using a term like non-linear science is like referring to … zoology as the study of non-elephant animals” — physicist Stanislaw Ulam ↩︎
17. Known as Charles’ law ↩︎
18. A contention I’m making is that, under the hood, this is what a lot of data science in politics boils down to ↩︎
19. You might ask, why not just average your RCTs across time to cancel out temporal variability. This does help in that it gives you “universal” effects that are valid across time, but the challenge is that, as seasoned operatives will tell you, the key to an effective campaign is responding in the moment ↩︎
20. Another book that has really informed my perspective, Steadfast Democrats by Ismail K. White and Chryl N. Laird, convincingly demonstrates the importance of network effects in explaining Black Americans’ continued overwhelming support of the Democratic Party ↩︎
21. And/or if you disagree that these complications are relevant to the work of political data! ↩︎