In this intellectual context, an article
magazine that included many prominent authors with RCT evidence showed that an intensive (hence expensive), multi-faceted (hence complex design), program for the poor—when implemented well—delivered income gains in five of six countries was doubly important. First, it showed it was possible for in-kind anti-poverty programs to be as effective and even, perhaps, a bit better than, cash in spite of the higher costs. Second, it showed that RCTs were not just useful in perhaps preventing people from wasting scarce time and money on efforts that were ineffective by showing negative results (which, while useful, is hardly the most popular outcome) but also in finding and demonstrating that at least something was effective (and even borderline cost-effective and generated positive net present value [NPV] at a moderate discount rate).
Evaluating interventions: it pays to be ignorant
As I pointed out many years ago, one downside of rigorous evaluation for advocates of particular approaches is that, while it may prove the gains are positive, it also pins down the upside of the claims that can be made. This implies that for people promoting otherwise popular and appealing interventions that “it pays to be ignorant,” as not having any rigorous evidence leaves advocates free to promote potentially persuasive (to donors, if not people who are sceptics) narratives of huge upsides, lives transformed by a simple and cheap intervention.
And as I point out in the recent CGD blog post, “The least you can do is better than the best you can do” and as I elaborate more fully in a recent CGD working paper (based on a speech I gave in December at Michigan State University as part of a symposium jointly sponsored by Reason Foundation), the gains from this targeted intervention are just dwarfed by the gains from labor mobility. Averaged across the five study countries the multi-faceted program cost $4,545 over two years to produce $344 in year 3 benefits. Averaged across those same five countries the (lower bound) on the annual wage gain for a typical low skill worker of moving to the United States is $13,119—almost 40 times larger. So the gains from a “state of the art” design and rigor of evidence anti-poverty program might be positive, but they are also very very small relative to just allowing workers to move to higher productivity places–and require large costs to achieve.
But there is a second point in the recent paper. One of the big debates in development is how much research effort should go into broad “national development” topics like how to generate sustained economic growth and how much should go into the rigorous evaluation of specific interventions. For the last decade or so the randomista crowd has appeared to have gained the upper hand by heaping contempt on the “growth regression” research and by suggesting it is better to have firm, internally valid, estimates of the LATE (local average treatment effect) of specific programs than to do research on big picture questions on which the same level of precision and confidence will never be achievable. As I have argued recently at NYU they have been mostly wrong about the practical usefulness of LATE estimates, but here I wanted to make a separate argument about the relative magnitudes, now that the RCT movement has pinned down some upper bounds on income impacts.
A simple calculation
Let’s do a super simple calculation, one that is super favorable to the benefits of the graduation program. If we take the average across the five countries then $4,545 in costs generated $344 in year 3 benefits. Suppose we assume those benefits are sustained forever then their NPV at a 5 percent discount rate and we translate to per person on the assumption of a four person household and we round down the costs from $4,500 to $4,000 then $1,000 in investment produces $1,720 (=(344/4)/.05) in benefits per person. Take Ethiopia, which receives about $4 billion in ODA per year. Suppose one fourth of that total were devoted to this program (and it could be scaled at constant efficacy to this amount) then a billion would produce 1.7 billion in NPV. Since there are 104 million people in Ethiopia this implies that $1 billion devoted to the “best case” targeted intervention would produce about $16.54 per person in NPV gains. That is not bad, its good—much better than many, many ways in which a similar amount of ODA or government resources could have been wasted or squandered on ineffective interventions.
On economic growth: gains and losses
But how does it compare to gains from economic growth? In a paper with three other authors (Kunal Sen, Sabyasachi Kar, and Salim Raihan) titled “Trillions Gained (and Lost),” we estimate the total NPV gain (or loss) per person from various growth accelerations (or decelerations). The paper describes the technical methods in detail but conceptually it is simple: we add up the (discounted) incremental GDP for each year of a growth episode relative to a plausible counter-factual of what GDP would have been without the acceleration or deceleration,.
Figure 1 (which is Figure 5 from the CGD working paper) below shows the results for selected of the larger gains and losses from growth episodes. For instance, the method says that Indonesia had a growth acceleration in 1967 and that growth episode lasted until 1996 and, relative to a plausible counter-factual, non-growth-acceleration, growth rate the NPV gain was $9,712 per person. In contrast, the method says that Mexico has a growth deceleration starting in 1981 that lasted until 1989 and the loss in GDP per person from that episode (a “lost decade”) was $10,811.
The obvious point is that the well-known large, extended growth episodes (Korea 1962, Taiwan 1962, Singapore 1968, Chile 1986, Poland 1991) produced gains in output per person that were near 1000 times larger than the per person gains of devoting a billion dollars to the state of the art anti-poverty program in Ethiopia. Similarly, the large growth slowdowns (Brazil 1980, Cote d’Ivoire 1978, Venezuela 1977 and 1985, Algeria 1979) produced losses per person (relative to counter-factual) that were also near 1000 times larger. Even “modest” per person growth accelerations like India’s in 2002 are 100 times bigger.
The obvious point is that if one wants to use rigorous evidence to establish a lower bound on the impact of a program one has to live with the fact this establishes an upper bound. So if the precision of estimation allows the researchers to just reject the hypothesis the year 3 impact is $344 then one can also reject that the impact is larger than $688. So the gains of $17 per person (which is more generous than the actual numbers from the paper) can be no higher than $34 per person. So the most optimistic claim one could make is that the impact of the targeted intervention is not 940 times smaller than the gains from the Korean 1962 growth episode, it might be only 470 times as small.
A little perspective
As we get into these ratios of hundreds to one and thousands to one it is hard to get perspective.
For instance, I am 5 feet 10 inches tall and the Empire State building is 1,250 feet tall—so it is 214 times as tall as I am. This means I am about four times taller compared to the Empire State building than is the impact of a billion dollars on per person NPV in Ethiopia on a state of the art anti-poverty program is large relative to the per person NPV gain from the growth acceleration in Korea in 1962.
Vietnam’s NPV per person gain from the growth acceleration in 1989 just until 2010 (when our data ended) was twice as big relative to the anti-poverty program per person gain (418 to 1) as the Empire State building height is to my height (214 to 1). Suppose the anti-poverty program gain was at its upper limit of how big it could be, $34 per person, then the ratio of gains would be just the same as Empire State building’s height to mine.
Here is another analogy to give perspective. The distance from my home in Cambridge to my office at Harvard is about 4 miles. The per person NPV gain from the growth episode in Thailand from 1958 to 1987 is the same ration to the hypothetical per person Ethiopia gain as the distance to Los Angeles to my house is to the distance from my house to my office. Suppose all four billion of ODA to Ethiopia could be devoted to the program at twice its estimated effectiveness ($34 per person), then the Thailand growth gain to anti-poverty program gain is only about the same ratio as my house to Rochester, NY as my house to my office.
Figure 1. The per person gains and losses from large growth episodes are 100 to 1000 times larger than the gains per person from devoting a billion dollars to a state of the art anti-poverty program in Ethiopia
Source: Pritchett (2018), Figure 5, based on Pritchett, Sen, Kar and Raihan 2016.
Table 1. Comparing the size of this and that: Gains or losses in per capita income from growth episodes to the size of the total gain from a billion dollars to an anti-poverty project compared to other ratios of this to that
|Measure (units)||This||Size of this||That||Size of that||Ratio of the sizes of this and that|
|NPV per person (dollars)||India’s 2002-2010 growth acceleration (gain)||$2,426||1 billion to anti-poverty program in Ethiopia||$17||147|
|Height (inches)||Empire State Building||15,000||Me||70||214|
|NPV per person (dollars)||Vietnam’s growth acceleration (gain)||$6,914||1 billion to anti-poverty program in Ethiopia||$17||418|
|NPV per person (dollars)||Peru’s 1981 growth deceleration (loss)||-$8,104||1 billion to anti-poverty program in Ethiopia||$17||490|
|NPV per person (dollars)||Indonesia’s 1967 growth acceleration (gain)||$9,712||1 billion to anti-poverty program in Ethiopia||$17||587|
|Distance (miles)||My house to Disneyland||2,980||My house to HKS||3.9||764|
|NPV per person (dollars)||China’s growth accelerations (1977 and 1991 combined)||$12,936||1 billion to anti-poverty program in Ethiopia||$17||782|
|Weight (lbs)||African elephant||13,000||Miniature Poodle||16||813|
|NPV per person (dollars)||Korea’s 1962 growth acceleration (gain)||$15,941||1 billion to anti-poverty program in Ethiopia||$17||964|
|NPV per person (dollars)||Venezuela’s growth decelerations (1977 and 1985 combined)||-$24,386||1 billion to anti-poverty program in Ethiopia||$17||1474|
|NPV per person (dollars)||Brazil’s 1980 growth deceleration (loss)||-$61,353||1 billion to anti-poverty program in Ethiopia||$17||3710|
Source: Pritchett et al (2016) for magnitude of growth episodes, author’s calculations based on results from Science paper for anti-poverty program, Google searches for the rest (except my height, which I exaggerated).
One could argue…
At this stage, there are a couple of responses. The most common: “Of course everyone knows the potential gains from a large growth episode are orders of magnitude larger than the potential gains of targeted programs but ‘we’ (as trained economists, or trained development practitioners) don’t know how to help a country create a growth episode and hence comparing something we know how to do with some degree of certainty and reliability with something we don’t know how to do is pointless.”
The obvious response to that objection is two-fold. One, if it is really, really, important and we don’t know how to do it, that seems like a really compelling argument for a lot more not less research to be devoted to the topic. Two, “we” don’t need to come to certainty or even high probability of getting right not wrong answers. Given the enormous magnitudes even small gains in the likelihood of producing a large positive (or avoiding a large negative) growth episode have massive expected value. If the stakes are trillions of dollars even a tiny increased in the likelihood of producing a win is of massive value.
From this point the arguments against research focused on the big picture questions and on rigorous estimates of how to, at best, produced modest sized gains have to get increasingly strident.
One could argue that, in spite of the massive gains from knowing the future (as well illustrated by Biff’s possession of a future sports almanac in Back to the Future II), little research by reputable physicists is focused on faster than light travel because our best physical theories say it is impossible. But no one believes a priori economic performance is invariant to human decisions.
Alternatively, one could argue that research would have not just small, but exactly zero impact on getting answers that would improve economic performance. While it is easy to show that growth research is unlikely to produce certain, completely reliable, and universal answers, the argument that any progress is impossible is very hard to entertain.
It’s about ideas
Finally, one could argue that decisions about actions that matter for growth are impervious to evidence—as say, they are fixed by ideologies that are impermeable to evidence or are completely structured by “interests”—and hence ideas don’t matter. But I tend to side with Keynes that, in the end, it is roughly only ideas that matter. The fact that some of the world’s largest growth accelerations are from Communist parties adopting more capitalistic and market responsive approaches at the very least suggests that policy stances and actions are actually quite flexible. Many growth episodes can be (roughly) dated to instances in which political actors announced the adoption of new approaches because of new ideas, so the argument that research could not, in principle, affect the course of events in the world seems implausible at best.
This post was first published on The Center for Global Development Blog
Lant Pritchett is one of the authors of the ESID book Deals and Development