Reinventing Foreign Aid Essay #4

This is the fourth summary and commentary on the essays in Reinventing Foreign Aid

It Pays to Be Ignorant: A Simple Political Economy of Rigorous Program Evaluation

By: Lant Pritchett

Dr. Prichett lays out a simple linear model that attempts to describe why so few aid programs ever receive rigorous evaluation using randomized trials. He assumes, based on a dozen years at the World Bank, that any new intervention has a set of rational supporters whose primary goal is to see their project receive as much funding as possible. As such, they must convince three groups to support the intervention: (1) core supporters, who have a low threshold for necessary efficacy because of either altruism or a personal stake (such as a production or delivery contract) in the intervention; (2) a "middle" group that reflects the general public, who are asked to give up their own wealth to pay for the intervention; and (3) hard-headed economists who have fairly high requirements for the efficacy of the programs they support and will only accept scientifically sound, randomized trials as evidence.

The "savvy activist" with enough money to start a program must decide between conducting a rigorous trial or the classic "pilot and promote" approach to convincing people to support her. The decision of which to do depends on a number of factors, most importantly the activist's best guess as to the actual effectiveness of the intervention and the general public's level of altruism. For a very effective program that yields large marginal gains for additional money, such as vaccinations or Mexico's "PROGRESA" program, it makes sense to perform a rigorous evaluation. For many aid programs the benefits are harder to see in a rigorous evaluation, and so the largest gains in support come from spending resources promoting the results of a pilot program. If one assumes a relatively low level of altruism among the general public, then almost no programs will meet its standards for efficacy as measured using rigorous evaluations. Thus, the most rational choice for nearly all aid advocates is to spend evaluation resources on advertising, since promotional activities increase both public interest in aid and that particular program.

The last third of the article is devoted to the very strange relationships that can develop between coalitions of savvy activists pushing different interventions and objectives. Tacit agreements often arise in such cases to prevent widespread publication of unfavorable rigorous test results, although the promoter of a more effective program might try to undercut others' support by leaking results. However, this "betrayal" inevitably comes back to haunt the leaker, and so a sort of code of silence develops and the message expressed is that aid is good and the world needs more of it.

This essay is my favorite so far in the book. I would absolutely love to get Dr. Pritchett and Nassim Nicolas Taleb together for pie and coffee sometime. I think Dr. Pritchett is a little too excited by the possibilities of models and graphs, but he includes a compelling narrative to explain how the aid community views evaluations. Dr. Taleb is perhaps the best living antidote to intellectual hubris, having spent much of the last few years predicting the financial crisis by pointing out the inherent flaws of trying to impose mathematical rules on things as fickle as people. Both seem to share a passion for exploring how people live in an uncertain world by embracing that uncertainty rather than trying to hide it behind equations or grand ideas. Anyone who can help me arrange such a meeting gets a slice or two, possibly even a whole pie to take home. Plus, I make coffee in a French press, and have a bean grinder that's waiting for a worthy occasion.

In a previous post, I suggested that randomized trial evaluation is a tool for agencies such as the World Bank, USAID or CARE that need to generate substantial public support, including senior bureaucrats and academics who are, presumably, hard-headed economists. To the extent these organizations need a way to determine which "Seekers" to reward, the more their internal funding allocation is governed by hard-headed economists the better. But in order to pay them, and engage in world-wide development and aid efforts, a large public outreach is required. As such, it is in the interest of program advocates to find powerful spokespeople such as Bono or Madonna.

Smaller organizations that rely on core supporters alone are much more free to implement any intervention they have developed. If that program is, say, a school for slum-dwellers from Nairobi or an NGO that delivers medical supplies directly to hospitals in war zones, then the good done is obvious to the few altruistic supporters needed. In that case, associations with other aid organizations should be approached with caution, since a larger organization has a built-in mandate to do the most good possible with its budget, rather than accomplish the goal of the smaller organization. For example, maybe the school should accept some nearby villagers instead of just kids from the slums, who tend to be drug and behavior problems, thus raising test scores and the partner's efficacy rating. Or perhaps the medical NGO could carry more than just life-saving supplies and run a vaccine dispersal while it's in country, since vaccines are good and the larger partner is under pressure to promote them everywhere.

The enemy of all Planners is accountability because few programs scale very well, while Seekers must embrace it because their smaller support base is often personally involved, if only to do things like fundraising or explaining where the organization fits in the aid/development/relief world. The World Bank seems to have employed two of its fiercest critics who have published literally volumes about the need for it to change dramatically with little practical effect. A small NGO, on the other hand, needs to keep its core supporters excited and involved with a constant sense of purpose, otherwise it has to shut down. This narrative-instead-of-quantitative accountability does not scale well, but then very few programs do, so how should one evaluate those that stay small? The next essay in the book discusses how this problem ought to be framed in development circles, where the Planned "solution" has often become the problem.

However, the need for an independent, small-scale evaluator who travels with little more than clothes, a baking mat, rolling pin and pie plate is increasingly obvious. I'd love to start or support a video blog along the lines of the Hulu show Cooking up a Story devoted to traveling to places that receive aid and seeing if it is getting easier to make pies. While a good postmodernist might reject "scientific" evaluation and master narratives, pastries transcend such philosophical issues and get to the heart of whether the program is doing good. At the very least, a poorly performing program would have a hard time explaining why I should should stop making pies (or maybe waffles).