While the COVID-19 pandemic underscores the centrality of good data to sound public policy making, the pandemic's more subtle lesson may be about the frequent elusiveness of truly meaningful data.
Indeed, the difficulty of determining both what to count and how to count it has been with us since the pandemic's earliest days. Initially this meant questions of how to measure cases of illness and model the course of disease spread. Subsequent discussions have focused on evaluating racial and ethnic groups’ differential disease burdens, assessing case positivity and fatality rates across space and time, and analyzing the evidence surrounding a range of public health interventions, medical treatments, and vaccines. Following the introduction of successful vaccines, even the question of what to count as a meaningful infection rests unsettled (Khullar 2021).
All of which is to say that Deborah Stone's important new book, Counting: How We Use Numbers to Decide What Matters, arrives at an apposite moment. Stone's argument for what one might call mindful counting or conscious and intentional quantification—“every counting rule is a human judgment, ours to make and ours to question” (91)—certainly resonates during a public health crisis where issues of counting are paramount. But the book also shows the historical pervasiveness of such questions, as when we look back and consider, for example, the construction and impact of the US Constitutional Convention's three-fifths compromise. Given its broad sweep and applicability, Stone's book provides essential insights for practitioners and students of public policy. All should read Counting early in their careers and educations—and then revisit the book to be reminded of and reinvigorated by its lessons.
Stone's lessons about how numbers “are always products of human judgment” (61) are distinctive from and also complement those in, for example, Darrell Huff's timeless 1954 book, How to Lie with Statistics. Where Huff focuses on common and often deliberate practices of statistical misinterpretation and misrepresentation, Stone is less interested in statistical analysis as such. Rather, Counting chronicles the social processes by which data are constructed and the mechanisms by which social meanings are derived from statistical information. Where Huff (1954: 8) emphasizes individual-level motives and assorted actors’ “bumbling and chicanery,” Stone's analysis centers on social and institutional processes.
Each of the book's six crisp chapters offers a distinct argument and provides clear demonstrations of how we do—and how Stone believes we should—use numbers: “Numbers grow from human imagination. We shouldn't put forth numbers or rely on them without examining the human judgments they stand on” (218). Rather than a prescriptive step-by-step toolkit, the chapters provide ideas and examples that the reader can extend or apply to their own interests. Each chapter's argument is illustrated with a range of cases from across the social sciences and political history along with helpful cartoons, as in Stone's widely read classic, Policy Paradox: The Art of Political Decision Making (2011).
Chapter 1 argues that, quite simply, “there's no such thing as a raw number”—all numbers are socially constructed. By this Stone means that numbers are unavoidably human creations, and that the processes underlying their production and use merit our attention. The chapter demonstrates the argument through descriptions of, among other things, the processes of constructing Bureau of Labor Statistics measures of unemployment and US Census Bureau measures of race and ethnicity (“censuses convert the mix of beliefs, practices, rules, and rights into artificial categories with seemingly clear boundaries” ). The chapter's admirable political evenhandedness allows it to show both the problematic racial implications of census categories that effectively preserve a “one-drop rule” and the valuable ideas underlying Donald Trump's criticisms of some measures of the unemployment rate. (Though Trump quite never put it this way, especially before he became president he criticized the standard reliance on U-3, which measures only unemployed workers, and advocated for using U-6, which adds discouraged and underemployed workers.)
Chapters 2 and 3 interrogate “How a Number Comes to Be” and “How We Know What a Number Means.” The second chapter illustrates the complexities of measure creation through case studies of scholarly efforts to build the global Varieties of Democracy index and United Nations efforts to construct global measures of gender inequality that include countries with a range of understandings of gender and dissimilar norms, practices, and laws. The chapter also describes the processes of creating predictive algorithms for crime and recidivism like PredPol and COMPAS, examples that are among the book's most compelling. As Stone shows, if some areas are unjustly overpoliced, this can produce high crime numbers that a predictive algorithm then uses to send more police to those same areas, yielding a “runaway feedback loop” in which “the algorithm causes the numbers it counts to come into being” (60; emphasis in the original). The third chapter, on how we use narrative and contextualization to interpret and understand numbers, recalls Stone's classic article on “causal stories” (1989) and discusses child mortality, poverty, GDP—which “doesn't distinguish between businesses that enhance our quality of life and ones that spoil it” (86)—and the lead water crisis in Flint, Michigan, the story of which “shows how the weak can wield numbers against the strong” (97) when advocates use data to document and publicize injustices. Some cases are elaborated at greater length than others; whether one is already familiar with a case or learns about it for the first time here, the cases are engagingly presented.
Chapters 4 and 5 illustrate “How Numbers Get Their Clout” and “How Counting Changes Hearts and Minds.” Chapter 4’s interrogation of the preferences and interests underlying measurements is particularly effective in outlining why the US Constitutional Convention arrived at the three-fifths compromise in 1787 and why using crime prediction algorithms can reproduce and exacerbate existing inequalities; that said, its critique of teacher-value-added measures is somewhat less persuasive. Likewise, chapter 5’s analysis of how measures change our perceptions and actions is most compelling in its discussions of health monitoring devices (10,000 steps!), the construction of census racial categories, and the growth of the Innocence Project, though it is somewhat less so in its discussion of the American National Election Study. Particular examples will resonate more or less depending on the reader, but Counting consistently provides illuminating explanations of thorny issues.
Chapter 6, “The Ethics of Counting,” moves from trolley problems, cost-benefit analysis, and debates about utilitarian versus deontological ethics to SAT adversity scores and automated employee evaluation metrics. These cases collectively show that, because “whenever we measure, we have to make trade-offs” (201), it is important to evaluate trade-offs using sound normative guidelines. The epilogue tackles the COVID-19 pandemic, including a discussion of former New York Governor Andrew Cuomo's once-lauded presentations of public health data. The fact that Cuomo subsequently resigned, in part, for manipulating nursing home death data—he should have followed Stone's admonishment: “If I have one message about counting, it is: Stay humble” (217)—actually serves to substantiate Stone's broader point about the social production of data and the necessity of seriously interrogating statistical information.
The book's probing critical analytic lens may leave it somewhat open to misinterpretation. While Counting does not quote Robert F. Kennedy's famous dictum that GNP “measures everything in short, except that which makes life worthwhile,” some readers may find that the book sometimes implies a similarly deep skepticism. Readers who focus on claims like “poverty is many things, but it isn't a number” (72) and “social issues aren't math problems” (138) may conclude that Counting is distrustful of the very act its title describes and would prefer our generally eschewing measurement and data altogether.
But it would be a mistake to read Stone as arguing against quantification. Rather, the book advocates for approaching data with reasonable skepticism, sufficient interrogation, and transparency and humility: “We can't do without numbers, so the challenge is how we can do a lot better with them” (xv; emphasis in the original). I highly doubt, for example, that Stone would laud Greece's lack of an agency like the Congressional Budget Office before the 2009–10 onset of its debt crisis (Lewis 2011: 48).
Any potential for such misreading might be allayed by considering positive examples where mindful counting has aided scientific truth and social progress, like Stone's discussions of a Dartmouth undergraduate student whose research helped show COMPAS's shortcomings (109) and the Innocence Project's efforts to free those wrongfully imprisoned. Such positive examples abound, though some may not initially appear as such. Take, for instance, the notorious hydroxychloroquine episode from the pandemic's early days. In this case, in March of 2020, the FDA used highly problematic data to authorize an ineffective drug for seemingly political reasons. But fewer than three months later, by June of 2020, hydroxychloroquine had gone through a range of processes described in Stone's book: a broad set of experts interrogated, studied, and refuted the earlier claims, after which the authorization was ignominiously withdrawn (Thomson and Nachlis 2020). Or take the FDA's August 2020 authorization announcement for convalescent plasma, which involved the agency's commissioner, at a press conference the night before a political convention, overstating the treatment's efficacy and mistaking a relative rate of change in a subpopulation for an absolute effect size across a full sample. Although this, too, was a serious statistical error, experts swiftly pointed it out, the media reported experts’ findings to the public, and the commissioner corrected the error the very next day (Nachlis 2020).
Moving beyond the public health context, another positive example Stone marshals to great effect is GDP. While “changing a measuring stick to be more fair is easier said than done” (189), critiques of GDP are now so widely recognized that, as the book notes, alternative measures of social progress and living standards are being adopted by a number of governments and organizations. It is also worth emphasizing that it is now standard for much social and economic analysis to include distributional data alongside topline aggregates. That major media outlets now report on issues like median income stagnation, and that social movements and social scientists have pushed ideas of “the 1%” and “the 99%” into national discourse alongside GDP, reflect precisely the kinds of conscious quantification and critical faculties that Counting extols (Byrne 2012). Movements for racial justice like Black Lives Matter have likewise effectively combined the experiences and the data surrounding racialized state violence to catalyze a global social movement (Francis and Wright-Rigueur 2021).
Counting's positive upshot also extends to social science research, as the book's arguments can help correct some of contemporary social science's less constructive tendencies. Whether the problem is quantification that seeks statistical but not substantive significance, inattentiveness to construct validity, a focus on hypothesis testing that tends to sideline null results and noncausal quantitative description, or p-hacking and the replication crisis, Stone's book can help address a range of longstanding issues that the age of MTurk has plausibly exacerbated. Ideally the book can help future analysts merge understandings of data's social context with skilled statistical analysis and deep data literacy. And perhaps the prominence of replication crisis discussions provides evidence that Stone's ideas are increasingly ascendant. There is even a new Journal of Quantitative Description, many of whose goals overlap with Stone's (Munger, Guess, and Hargittai 2021).
With respect to using Counting to engage students and policy makers, because the book principally illuminates the stakes of measurement and not the methods and concepts underlying data analysis, it might be effectively paired with a primer on statistical concepts, like Charles Wheelan's Naked Statistics (2013), an introduction to statistical analysis like Kosuke Imai's textbook (2018), or a salient methodological case study like Westwood, Messing, and Lelkes's analysis of probabilistic horse-race election coverage (2020). Substantively, Counting might be usefully paired with an extended case study like Melissa Nobles's Shades of Citizenship (2000) or Caroline Criado Perez's Invisible Women (2019). For students of health politics and policy, Counting might be taught alongside Jeremy Greene's Prescribing by Numbers (2007), Miriam Laugesen's Fixing Medical Prices (2016), Daniel Carpenter's discussion of “conceptual power” in Reputation and Power (2010), or Welch, Schwartz, and Woloshin's Overdiagnosed (2011). Further afield, for more arts-oriented students, the book might support a discussion of the sheer banality of Rotten Tomatoes's reduction of all film reviews to a set of dichotomous scores converted into a 100-point scale. Such suggestions merely serve to demonstrate the book's wide-ranging applicability in a variety of settings where it can either center the analysis or enhance an analysis focused elsewhere.
More broadly, readers should find it helpful that Counting provides the arguments underlying a seemingly growing impulse to question and interrogate data. This volume offers a clear analytic framework for organizing and catalyzing such thinking. Deborah Stone's important new book may well herald a future with counting that is more honest, more ethical, and more meaningful.