Impact Evaluation of Programs that Address Global Environmental Change
April 20-21, 2010
Lamont Campus, Columbia University
Contact: Malgosia Madajewicz
Workshop Program (167KB)
|Welcome Address||Steve Zebiak|
|Workshop Overview (2.66 MB)||Malgosia Madajewicz|
|Challenges in evaluating impact of global environmental programs (518.9kb)||Aaron Zazueta|
Session 1: Statistics and case studies: Broad criteria for choosing between different approaches to impact evaluation
Session Chair: Fred Carden
|Fitting the key to the lock - addressing the nature of the intervention and the specific evaluation task (500.2kb)||Patricia Rogers|
Session 2: Beyond average impacts of programs: Combining evidence from large n and small n studies
Session Chair: Uma Lele
|What works for whom in what circumstances - methods for complicated and complex interventions (1.13 MB)||Patricia Rogers|
|Impact evaluation of weather index insurance for smallholder farmers (131.42kb)||Osvaldo Feinstein|
|Till time (and contexts) do us apart: The heterogeneous impacts of programs in longitudinal settings (694.2kb)||Sanjeev Sridharan|
|Six underappreciated challenges to impact assessment for n's of all sizes (276.95kb)||Macartan Humphreys|
Session 3: Learning about program impacts when we do not have large n data
Session Chair: Fred Carden
|Impact evaluation for small n (485.55kb)||Fred Carden|
|REDD: Design, implementation and evaluation challenges (1.16 MB)||Uma Lele|
|Understanding causal relationships with case studies: possibilities and limitations (124.16kb)||John Gerring|
|Accessing causality in participatory projects: How do we know that climate forecasts affect farmer decisions? (10.48 MB)||Carla Roncoli|
|Evaluating the contributions of the GEF to the South China Sea (1.24 MB)||Aaron Zazueta|
Contributed Documents, Links, and References
- Evaluation checklists
- CIPP checklist (includes specific suggestions on "context")
- Sourcebook for evaluating global and regional partnerships programs - Principles and Standards
- Process documentation research
- Network Of Networks on Impact Evaluation (includes,among other publications, the "Impact Evaluation and Development: NONIE Guidance on Impact Evaluation")
- Book on evaluating climate change and development
- Knowledge Utilization, Diffusion, Implementation, Transfer, and Translation: New Directions for Evaluation, No. 124, Winter 2009 Judith M. Ottoson (Editor), Penelope Hawe (Editor
- Food Security for a Billion Poor (Editorial, 26 March 2010, Volume 326, Science Magazine)
- Transforming Agricultural Research for Development Presentation (Global Conference on Agricultural Research for Development (GCARD) Montpillier, France, 2010)
- Transforming Agricultural Research for Development (TAR4D), A Global Consultants' Report (GAT Report for GCARD 2010 - Version 11.0.
- M.S. Swaminathan Research Foundation at 21: Report of the Independent Program Review
Speakers and Panelists
- Fred Carden
- As Director of the International Development Research Center's Evaluation Unit, Fred Carden is an expert on developing innovative ways to assess the impact of development programs and the contributions of research to concrete policy and behavioural changes. Carden has been with the Evaluation Unit since 1993. His work has focused on assessing the influence of research on public policy, developing tools to assess organizational and program performance, and outcome mapping — a people-focused approach that evaluates development programs based on how they affect relationships, activities, and behaviour. He has written extensively on evaluation, international co-operation, and environmental management. He has taught at York University in Toronto as well as in Tanzania and Indonesia. Carden holds a PhD from the Université de Montréal and a master's degree in environmental studies from York University.
- Osvaldo Feinstein
- Professor at the Madrid Complutense University's Master in Evaluation and advisor to the Spanish Evaluation Agency. Senior consultant with the World Bank, and the evaluation offices of UNDP and the International Fund for Agricultural Development (IFAD). During 2009 senior consultant with the United Nations Evaluation Group. Former advisor and manager at the Operations Evaluation Department, and senior evaluator at IFAD. Published and edited several articles and books on evaluation, development and economics. Former consultant with the Inter-American Development Bank, the African Development Bank, the UN Technical Cooperation Development Programme, the Latin American Institute of Economic and Social Planning, UNESCO and the ILO.
- Alan Fox
- Alan Fox serves as Evaluation Adviser in the United Nations Development Programme Evaluation Office. He leads country studies and thematic evaluations, and has specific responsibilities overseeing the evaluation of UNDP support programmes in the environment and energy sectors. Alan joined UNDP in 2009 after previous positions in the US government and as a consultant. He has served as Special Assistant to the Mayor of Cleveland, Ohio (1985 - 1989), and Associate Assistant Administrator for Water at the U.S. Environmental Protection Agency (1989-1993). Subsequently, Alan moved to the private sector, as Vice President in the infrastructure division of the international engineering and consulting company Pöyry PLC (Helsinki Finland), and then in 2004 he established his own firm, Transboundary Consulting, LLC. Alan's portfolio of consulting and evaluation assignments spans over 30 countries, for a wide array of international agencies and private sector clients. Alan holds a BA (political science) from Case Western Reserve University and an MBA from the University of Pretoria, South Africa
- John Gerring
- John Gerring (PhD, University of California at Berkeley, 1993) is Professor of Political Science at Boston University, where he teaches courses on methodology and comparative politics. His books include Party Ideologies in America, 1828-1996 (Cambridge University Press, 1998), Social Science Methodology: A Criterial Framework (Cambridge University Press, 2001), Case Study Research: Principles and Practices (Cambridge University Press, 2007), A Centripetal Theory of Democratic Governance (Cambridge University Press, 2008), Concepts and Method: Giovanni Sartori and His Legacy (Routledge, 2009), Social Science Methodology: Tasks, Strategies, and Criteria (Cambridge University Press, 2011), Global Justice: A Prioritarian Manifesto (in process), and Democracy and Development: A Historical Perspective (in process). He served as a fellow of the School of Social Science at the Institute for Advanced Study (Princeton, NJ), as a member of The National Academy of Sciences' Committee on the Evaluation of USAID Programs to Support the Development of Democracy, as President of the American Political Science Association's Organized Section on Qualitative and Multi-Method Research, and is the current recipient of a grant from the National Science Foundation to collect historical data related to colonialism and long-term development.
- Macartan Humphreys
- (PhD (Government) Harvard 2003, MPhil (Economics) Oxford, 2000) works on the political economy of development and formal political theory. Ongoing research focuses on civil wars, post conflict development, ethnic politics, natural resource management, political authority and leadership and democratic development. He uses a variety of methods including survey work, lab experimentation, field experimentation, econometric analysis, game theoretic analysis and classical qualitative methods. He has conducted field research in Chad, Ghana, Haiti, Indonesia, Liberia, Mali, Sao Tome and Principe, Sierra Leone, Senegal, Uganda and elsewhere. A new series of projects underway use field experiments to examine democratic decision making in post conflict and developing areas. Recent research has appeared in the American Political Science Review, World Politics, Public Choice, the Journal of Conflict Resolution and elsewhere. He is a research scholar at the Center for Globalization and Sustainable Development at the Earth Institute.
- Uma Lele
- Uma Lele retired from the World Bank in 2005 as Senior Adviser in the Operations Evaluation Department (OED, now called the Independent Evaluation Group, IEG) of the World Bank, an arm that reports to the Bank's Board of Executive Directors. She is leading a Global Team of Authors (Eugene Terry, Eduardo Trigo and Jules Pretty) preparing a paper on the changes needed in the Global System for Agricultural Research for Development, and implications for regional and CGIAR research priorities, to be discussed at the Global Conference on Agricultural Research for Development (GCARD), in Montpelier, France, March 28th and March 30, 2009. At the World Summit on Food Security, November 16th to 18th 2009, in Rome, she participated in the Roundtable on Implementation of the Reform of Global Governance of Food Security. At the American Evaluation Association (AEA), she organized a session on November 13, 2009, on Evaluation Challenges of Climate Change and Avoided Deforestation as a Complex Global Public Good. She was core team member of the Independent External Evaluation of the Food and Agricultural Organization (FAO) in 2006 and 2007 with responsibility for the evaluation of FAO's work in the areas of food and agriculture, forestry, fisheries, statistics and emergency assistance among others. She continues to work in the areas of Global and Regional Public Goods, and the Changing Aid Architecture. In 2008 she completed Independent Evaluation of the Implementation of the World Bank's 2002 Forest Strategy and Forest Related Global Programs related to Reduced Deforestation and Degradation (REDD) in the context of Climate Change. She is contributing to the World Bank's Design of Regional Health Systems Strengthening and TB Support Project for the control of tuberculosis in Africa. She served as a peer reviewer for the Independent Review of the CGIAR in 2008. She participates in the Quality Assessment Reviews of the World Bank's Global and Regional Partnerships.
- Marc Levy
- Marc Levy is Deputy Director of the Center for International Earth Science Information Network (CIESIN), part of the Earth Institute at Columbia University. He also serves as an adjunct professor in Columbia's School of International and Public Affairs. He is a political scientist specializing in the human dimensions of global change. He is an expert on environment-security connections, environmental sustainability, drivers of emerging infectious diseases, and the effectiveness of international environmental institutions. His work has been supported by a large number of national and international agencies. He has served on a number of global environmental assessments and frequently advises national governments and international organizations on global change issues. Before coming to CIESIN in 1998 Mr. Levy had teaching appointments at Princeton University and Williams College.
- Patricia Rogers
- Patricia Rogers is Professor in Public Sector Evaluation at the Royal Melbourne Institute of Technology, Australia. She has worked in diverse areas of public sector evaluation for more than 20 years, including projects with the UNDP, the Harvard Children's Institute, the World Bank Institute, the Ministry of Finance (Malaysia), the Foundation for Advanced Study on International Development (Japan), the Public Service Commission (South Africa), the Institutional Learning and Change initiative of the CGIAR (Consultative Group on International Agricultural Research), AusAID, Changemakers Australia (a peak organisation focusing on social change philanthropy), and NONIE (the Network Of Networks on Impact Evaluation). Her current research focuses particularly on methods for the evaluation of complicated and complex interventions, including a forthcoming book Purposeful Program Theory (2010).
- Carla Roncoli
- Carla Roncoli is an environmental anthropologist and has worked in applied research and international development for over 20 years. Dr. Roncoli's research centers on the communication of climate information, integration of local and scientific climate knowledges, climate vulnerability and adaptation in agricultural systems, and environmental management in conditions of climate uncertainty, with a regional focus on Africa (Burkina Faso, Ghana, Mali, Kenya, Uganda). Her research combines a variety of methods, including participatory, qualitative, and quantitative approaches. Dr. Roncoli has worked for 15 years as a research scientist at the University of Georgia (UGA) in the context of interdisciplinary research projects funded by USAID, NOAA, USDA, NASA, NSF, and the World Bank. She has played a lead role in the Climate Forecasting and Agricultural Resources (CFAR) project, implemented in Burkina Faso between 1998 and 2007. She is also an active participant in the Columbia University-led Center for Research on Environmental Decisions and in the Southeast Climate Consortium (the RISA for the Southeast U.S.). Dr. Roncoli has worked for several years with the Sustainable Agriculture and Natural Resource Management Collaborative Research Support Program, also serving as its Associate Director. After 15 years at UGA, Dr. Roncoli has recently joined Emory University. Her work is published in interdisciplinary journals such as Climate Research, Climatic Change, Climate, Weather, and Society, Agriculture and Human Values, Agricultural Systems, Society and Natural Resources, and Land Degradation and Development.
- Sanjeev Sridharan
- Sanjeev Sridharan is Director of the Evaluation Program at the Centre for Research on Inner City Health and Associate Professor of Health Policy, Management and Evaluation at the University of Toronto. Prior to his position at Toronto, he was the Head of the Evaluation Program and Senior Research Fellow at the Research Unit in Health, Behaviour and Change at the University of Edinburgh. His primary focus has been on evaluation influence, evaluation methodology, health inequities, cardiovascular disease and other co-morbid conditions. His interests in the past few years has been in evaluating a set of interventions, called anticipatory care, that seek to disrupt the linkage between deprivation and poor health. These anticipatory care initiatives target the poorest areas of the United Kingdom to enhance reach and engagement with prevention activities amongst high risk groups. Sanjeev has worked on a wide range of evaluation projects in a number of fields, including evaluations of interventions in health systems reform, juvenile justice and delinquency prevention initiatives, substance abuse, and criminal justice systems reform. He is presently working with the China Health Economics Institute to build evaluation capacity in the health sector in China. He is developing a post-graduate program in evaluation in five South Asian countries. He is advising the Ministry of Health in Chile on utilizing evaluation approaches to redesign health policies in Chile. He has also developed evaluation strategies for organizations including National Health Service Health Scotland and the Forestry Commission in Scotland. Sanjeev has a Doctorate in Social Ecology from the University of California at Irvine, MA in Public Policy from Purdue University, and a Bachelor of Technology degree in Civil Engineering from the Indian Institute of Technology. He is on the Board of the American Journal of Evaluation and Evaluation and Program Planning.
- Maximo Torero
- Dr. Máximo Torero is the Division Director of the Market, Trade and Institutions Division at (IFPRI) International Food Policy Research Institute, IFPRI Coordinator for Latin America, and a leader of the Global Research Program on Institutions and Infrastructure for Market Development. In this capacity, he directs the activities of an IFPRI unit that conducts research, with special emphasis on M&E of infrastructure and rural development interventions in urban and Peri-Urban areas through the use of randomized experimental design. His experience extends to projects in Latin America, Sub Saharan Africa, and Asia. Dr. Torero had a unique expertise on impact evaluation on projects linked to market access, property rights and infrastructure provision of water, electricity, telephones and roads. On infrastructure, Dr. Torero has been involved in several projects to evaluate the impact of access to infrastructure and had also design and implement pioneering designs in which an experimental approach is used to investigate the extent to which the cost barriers preventing poorer households from connecting to the different types of infrastructure can be addressed. These studies had led to several policy impacts, especially on the complementary effects of access to infrastructure. On property rights, Dr. Torero has worked intensively on urban and rural titling and crop choices as a result of titling projects (see for example the reference to his work in "The Mystery of Capital Deepens." Economist, August 24, 2006). Finally, on market access, Dr. Torero has worked on impact evaluation of contract farming arrangements to access to dynamic markets for small commercial farmers, and the creation and sustainability of urban-rural market institutions. His work has also focus substantially on the impact of rural households to phones as an instrument to reduce their asymmetry of information when accessing to markets.
- Aaron Zazueta
- Aaron Zazueta, a social anthropologist, is a Senior Evaluation Officer at the GEF Evaluation Office since 2002. He is responsible for Impact Evaluation in for the GEF and for the production of the GEF Annual Performance Report. Prior to joining the GEF, Dr, Zazueta was a member of teams that evaluated the World Bank's 1991 Forest Policy and the application of the Bank's Indigenous People's Safeguard and conducted project evaluations for the World Bank. He played a key role in helping to develop the Inter American Development Bank's (IADB) "Strategy for Citizen Participation". Between 1990 and 1996, Dr. Zazueta was Regional Director for Latin America at the World Resources Institute, where he led a team of professionals working with community groups, NGOs and governments in the formulation of environmental policies in Latin America. Dr. Zazueta also served as the strategic and policy advisor to the Central American Commission for Environment and Development, the Tropical Forest Action Program in Mexico, Ecuador and Guatemala, the Ministry of Sustainable Development in Bolivia and the National Commission of the Environment in Chile, Peru and Ecuador. Subsequently, he was appointed by Vice President Al Gore on to a special commission to assist the Bolivian President to incorporate sustainable development into the policies and programs carried out during his administration. He co-chaired the technical commission that drafted the Hemispheric Agenda for Sustainable Development, ultimately adopted by thirty two heads of state of the Americas in December 1996. Aaron has more than 20 publications on issues related to citizen participation, environmental policy and sustainable development.