Measuring a Project's Success

Governments, development organizations and other entities spend billions each year on programs aimed at improving the lives of poor people in developing countries. But have these interventions actually made a difference? That's difficult to answer, says one IRI researcher, because many programs haven't been rigorously evaluated. She hopes to incorporate impact evaluations into some of IRI's climate-risk management projects.

"Science-based methods are used in impact evaluations to enable us to learn what worked, for what purpose, under what conditions, and why," says IRI economist Malgosia Madajewicz.

"Impact evaluations can help us understand, for example, how an early warning system for drought can improve child nutrition and thereby educational outcomes, or what mix of programs can help farmers adopt sustainable practices, manage climate risks and spur growth in agriculture."

Consider the use of crop insurance to hedge against drought or other disasters. "It's easy to say that this many people signed contracts, this many people received payouts. But it's much harder to say what effect the insurance had on outcomes we ultimately care about, such as improving living standards and reducing vulnerability to disasters," says Madajewicz.

The reason is that crop insurance isn't the only factor that determines any of the above. "It may be that people who sign contracts happen to be more entrepreneurial. They may have had good outcomes anyway," she says.

Teasing apart these factors means that evaluation can't be an afterthought. "The best assessments happen when they are planned carefully into the program or project from the beginning, something which most development and donor organizations haven't been doing," says Madajewicz.

In the last decade, innovations in statistical methods have allowed researchers to better evaluate impacts, to get better at identifying causal effects as opposed to correlations.

In 2006, the Center for Global Development published an important report, When Will We Ever Learn? Improving Lives Through Impact Evaluation.

The authors write that after decades of spending billions for social programs, "It is deeply disappointing to recognize that we know relatively little about the net impact of most of these social programs...No responsible physician would consider prescribing medications without properly evaluating their impact or potential side effects. Yet in social development programs...no such standard has been adopted. While it is widely recognized that withholding programs that are known to be beneficial would be unethical, the implicit corollary-that programs of unknown impact should not be widely replicated without proper evaluation-is frequently dismissed."

As a result of this report and others, there's now a big push to conduct impact assessments more systematically.

In an effort to incorporate such assessment into IRI's climate-risk-management projects, Madajewicz is organizing a seminar series to introduce evaluation methods to IRI staff and other members of the Earth Institute. "We'll illustrate the methods and examine case studies of how they were applied," she says.

"We are anxious to support this research and to bring impact evaluation into the mainstream of IRI's work, which, in the end, is targeted toward improving outcomes on the ground," says IRI Director-General Steve Zebiak.

Development organizations such as the World Bank, the U.K.'s Department for International Development (DfID) and many governments are taking impact evaluations more seriously in their process for allocating funds, says Madajewicz. Many policy makers now believe they can't afford not to carry out rigorous impact evaluations, since they can't afford to spend money on ineffective programs. "They can't make decisions based on anecdotal evidence or people's opinions of what works and doesn't," she says.

Two of Madajewicz's current projects, still under development, try to understand what types of climate information can improve climate risk management in the agricultural systems of Uruguay and Uganda. "Under what conditions do farmers use climate information, such as seasonal forecasts and rainfall trends," she asks, "and what policies make climate information most effective in improving livelihoods under different types of conditions?"

Conducting impact evaluation in situations in which climate influences the impacts of the project present special challenges. Ideally, one would need to compare farmers' livelihoods when farmers use the climate information and when they don't over a long period of time. Of course, waiting long periods of time isn't useful to policy makers, who need to make decisions in the short term.

"For the evaluation, we would need to make do with a few seasons of observations perhaps, and available historical records. The fundamental questions here is, How can we project what the effects of the intervention will be in climate scenarios we haven't observed?" says Madajewicz.

"Existing methods do not offer sufficient guidance. We need innovative, creative ways of combining theory and data in order to learn about impacts in circumstances which may not be random draws from the data we have."

Later this year, the IRI will bring together experts from both the research and practice side to take part in a workshop on impact evaluation which will advance the needed innovations in methods.