Sloppy Math and the Austerity Debate

In 2010, two Harvard economists, Carmen Reinhart and Kenneth Rogoff, circulated a paper demonstrating that GDP growth is negatively correlated to public debt (debt-to-GDP ratio to be more precisely).  Their paper was highly influential and has been used to support the global austerity agenda.  For instance, it was cited in the “Paul Ryan Budget,” p. 80.  But the problem is that the paper is rife of basic math errors, as pointed out in a recent paper by Thomas Herndon, Michael Ash and Robert Pollin of the University of Massachusetts, Amherst.  On April 19, Paul Krugman wrote an op-ed piece entitled “The Excel Depression” for the New York Times publicizing an embarrassing Excel error in Reinhart and Rogoff’s spreadsheet.  (You can see the Excel screen capture here; instead of AVERAGE(L30:L49), Reinhart and Rogoff entered AVERAGE(L30:L44), excluding Australia, Austria, Belgium, Canada, and Denmark from their calculation.)  A week later, Reinhart and Rogoff contributed their own op-ed piece; they suggested that their mistake was inconsequential and continued their claim that “growth is about 1 percentage point lower when debt is 90 percent or more of gross domestic product.”  In their online appendix, the chart below was shown to make such a point.


Source: Reinhart and Rogoff’s New York Times online appendix.

You can find extensive discussions over the Reinhart-Rogoff controversy on the internet.  Here I just want to highlight some of the technical issues involving merely elementary arithmetic.  In fact, Thomas Herndon, a PhD candidate, discovered RR’s Excel error through a course assignment.  Students were supposed to replicate the findings of a famous paper in order to learn econometric techniques.  As RR’s work involves essentially calculating the means and medians, Herndon’s professor almost didn’t let him take on the project.  (See this New York Magazine article and this Colbert Report interview.)

A casual reader might get an impression of downward slide in GDP growth based on the above chart, but a careful reader might have noticed that the chart compares medians and arithmetic averages.  Students who have taken an introductory statistics course know that medians and means (often loosely referred to as averages) can be quite different if the distribution is skewed.  Furthermore, means and medians without accompanying measure of how much the data is spread out (such as standard deviation or interquartile range) provide incomplete information.

I wrote to Professor Ash and he kindly supplied additional information.  He and Professor Pollin published a response to Reinhart and Rogoff in the Times, with a comprehensive technical supplement.  They also made the data public to allow a close examination.  To illustrate the effect of Reinhart and Rogoff’s Excel error, one can make a comparison between the New York Times chart and a corrected one based on the inclusion of Australia, Austria, Belgium, Canada, and Denmark.  This inclusion alone increases median GDP growth by 0.3 percentage points in the 90 percent and above public debt/GDP category.


Median figures for blue line are from the NY Times; median figures for red line are from Table 1 of technical supplement by Ash and Pollin, “RR calculation method but with corrected spreadsheet.” The Excel error alone is responsible for a difference of 1.9-1.6=0.3 percentage points in RR’s above 90% debt/GDP category.

It is unclear why RR excluded available data of earlier years for Australia, Canada and New Zealand.  By including these three countries, it further increases median GDP growth in RR’s highest debt/GDP category to 2.5%, which is only 0.4 percentage points lower than that in the next highest debt/GDP category.


Median figures for blue line are from the NY Times; median figures for red line are from Table 1 of technical supplement by Ash and Pollin, “recalculation with both corrected spreadsheet calculations and inclusion of Australia, Canada and New Zealand early years.” The difference in median GDP growth is only 2.5-2.9=-0.4 percentage points between 60 to 90% category and above 90% category.

While most economists acknowledged some correlation between high debt and low GDP growth, the above 2 charts illustrate that the claim “growth is about 1 percentage point lower when debt is 90 percent or more of gross domestic product” by Reinhart and Rogoff is based on sloppy math and is unsubstantiated.

In the Times online appendix, Reinhart and Rogoff stated that they “gave significant weight to the median estimates, precisely because they reduce the problem posed by data outliers.”  Yet two paragraphs later, when reporting their findings in a paper published in 2012 (joined by Vincent Reinhart), they gave the means only.  They compared the mean from 1800 to 2011, 2.3 percent, to the mean of Herndon et al. from 1945 to 2009, 2.2 percent.  If we just cherry pick numbers, let us compare the tainted result that Reinhart and Rogoff published in the Times with means from 2000 to 2009 only.  Contrary to Reinhart and Rogoff’s central claim, GDP growth in the over 90 percent public debt/GDP category has actually outperformed GDP growth in the 60 to 90% public debt/GDP category.


In this chart, the median figures for blue line are again from the NY Times, and the mean figures are from Table 4 of technical supplement by Ash and Pollin. Data from 2000 to 2009 contradict Reinhart and Rogoff’s claim.

But the above chart is not particularly meaningful, because the standard deviation of each GDP growth figure (as shown in Ash and Pollin’s supplement) is at least 0.3.  A more responsible way to present the means is the following graph, taken from their supplement.


Source: technical supplement by Ash and Pollin, Figure 2.

People who have working experience with data know that mean or median conceals a lot of information.  In this case, it is evident that there is a wide range of economic performance outcomes in a given category.  Using the file containing data from 1946 to 2009 (the basis of the New York Times line chart above), one can make a scatterplot.  It is apparent that the relationship between public debt and GDP growth varies significantly.


Source: The file RR-processed.dta posted in U Mass website. Working spreadsheet was provided by RR; the U Mass group corrected errors and cleaned up the data.

By now, you should have realized that the same data can be graphed in different ways, which can lead to a vastly different impression.  The Reinhart-Rogoff affair offers educators a great opportunity to introduce many other quantitative reasoning topics, such as which measures of central tendency to use, correlation does not imply causation (does high debt cause low growth, or the other way around?), and so on.  It is quite feasible to ask students to perform their own analyses and make their own inferences, and I strongly encourage you to do so.

This entry was posted in Uncategorized. Bookmark the permalink.

One Response to Sloppy Math and the Austerity Debate

  1. Pingback: Stencila – Spreadsheet-like Live Reactive Programming Environment | 神刀安全网

Leave a Reply

Your email address will not be published. Required fields are marked *