Impact assessment in complex development situations

Gauging the impact of development work presents many challenges, not least the one of attribution. It is difficult to demonstrate that particular programmes result in significant and lasting changes in the well-being of intended beneficiaries, when a range of agencies and external factors are involved. No single agency can realistically claim full credit for observed social changes, something that is equally the case for communication and information work in development. This key list provides key resources that address the challenges of assessing the impact of development interventions. There is a focus on the impact of information products, services and projects. An additional selection of materials looks at development impact from a 'network' evaluation perspective - by tracking changes in relationships over time, and comparing this to what might be expected from a project's objectives.

This list of key resources was compiled for the Exchange Lunchtime Discussion on The challenge of impact assessment in complex development situations. Source welcomes details of additional resources and accounts of how they are useful - please send these to sourceassistant@hi-uk.org.

Selected resources

Books, reports, etc

Evaluating capacity development : experiences from research and development organizations around the world

HORTON, Douglas
et al
2003

Expand view

This book is the result of the Evaluating Capacity Development (ECD) Project, begun by ISNAR in January 2000, which aimed to improve capacity development efforts in research and development organizations through the use of evaluation. It explains how the project used an action-learning approach, bringing together people from various countries and different types of organisations. Six evaluation studies were conducted over the course of three years: exploring capacity development in a rural development NGO in Bangladesh; towards strategic management in a Cuban agricultural research institute; understanding capacity development in a plant genetic resources centre in Ghana; assessing organisational change in an agricultural faculty in Nicaragua; strengthening participatory research capacities in a Philippines root crops research centre; and expanding capacities in a rural development institute in Vietnam. Chapter 1 provides background reading on the ECD project that gave rise to the book. Chapter 2 discusses basic concepts of organisational capacity, capacity development and evaluation. Chapter 3 addresses two fundamental issues: why managers should be concerned with organisational capacity development and why they should evaluate capacity development efforts. Chapter 4 discusses issues related to the 'how' of capacity development. Chapter 5 discusses partnerships for capacity development while 6 outlines approaches and methods for evaluating organisational capacity development. Chapter 7 discusses how to utilize evaluation processes and results to advance capacity development and performance in an organisation

Knowledge and learning in online networks in development : a social capital perspective

CUMMINGS, Sarah
HEEKS, Richard
HUYSMAN, Marleen
2003

Expand view

This paper examines the role of electronic networks in facilitating knowledge development and social learning in development, using the concept of social capital. This concept was developed to describe the functioning of groups and societies, and has been applied to online networks outside development. This paper examines three non-development approaches for examining social capital in online networks and communities. It uses elements of these approaches as a basis for a framework tailored to the analysis of online networks in a development context

Measuring the impact of information on development

MENOU, Michel J
1993

Expand view

This report documents an initiative that, through its methods and focus, aimed to provide a realistic starting point to understanding the impact of information on development. It consisted of a electronic conference, followed by a workshop through which ideas about information and evaluation were shared and built upon. The report weaves together the input of dozens of information users and providers, policymakers, information scientists, and others from the South and the North, and presents a preliminary assessment framework as a starting point to future work in this area

Monitoring and evaluating advocacy : a scoping study

CHAPMAN, Jennifer
WAMEYO, Amboka
January 2001

Expand view

This scoping study has attempted to identify and document how various agencies and institutions have approached the assessment of advocacy. It sets out a number of frameworks that look at similar issues from different perspectives and, instead of promoting one framework as the 'correct' one, allows the reader to pick and choose what elements are most useful to them. The work was limited in scale, and focused in particular upon the approaches of NGOs. The insights and ideas from this study will contribute to a three-year action research project to be undertaken by ActionAid and partners in Africa, Asia and Latin America.

Network perspectives in the evaluation of development interventions : more than a metaphor

DAVIES, Rick
November 2003

Expand view

This paper argues for the use of a network perspective in representing and evaluating aid interventions. Commonly used linear evaluation tools such as the logical framework (logframe) or problem tree selectively represent what an agency is trying to do, and then assess particular aspects of the intervention. Social network analysis describes social relationships which, the author contends, is what development is about. The wide range of methods and theories for network analysis means that there are lots of ways of thinking about and describing expected outcomes of interventions. Social network analysis is also very flexible in terms of the scale of the project, and can accomodate non-linear processes of change. This paper presents these five arguments for social network analysis, and then describes next steps for developing a coherent approach to evaluation based on a network perspective

Outcome mapping : building learning and reflection into development programmes

EARL, Sarah
CARDEN, Fred
SMUTYLO, Terry
2002

Expand view

Assessing the impact of the efforts of a single external development agency is problematic because changes in the well-being of beneficiaries occur as a result of a ‘confluence of events’. Outcome mapping is a dynamic methodology that looks at the contribution an agency has made at project, programme or organisational level to influence the processes leading to changes in the behaviour, relationships, actions and activities of people and organisations. According to the authors, development is about people relating to each other and their environment. Outcome mapping is therefore a monitoring and evaluation system for current and completed activities and provides a framework and vocabulary for understanding changes and assessing efforts. It is based on principles of participation, iterative learning and evaluative thinking throughout.
This book includes a thorough explanation of the outcome mapping approach, and provides detailed information on workshop design and facilitation, as well as numerous worksheets and examples

Participation, relationships and dynamic change : new thinking on evaluating the work of international networks

CHURCH, Madeline
et al
2003

Expand view

Gives an overview discussion of the key characteristics of networks, noting that international development increasingly takes a network form. Networks typically put an emphasis on: facilitative leadership; building relationships and trust; light co-ordinating structure that allows decentralisation, autonomy and voluntary participation. However, these are charactersitics that traditional evaluation approaches have not been developed to address. Provides a useful check-list for evaluating networks and suggests some useful practical tools to approach the evaluation of networks, such as: ‘Contributions assessment’ to guage how effectively the network facilitates the circulation of resources, and enables people to make the contribution that they are capable of; ‘Clarification of aims and activities’ (adapted Weaver’s Triangle) tool to clarify how participants perspectives and activities diverge or converge over time; ‘Channels of Participation’ tool to assess how and were members interact in a network and look at changes over time; ‘Monitoring the edges’ to track independent networking stimulated but not through the centre

Streams of knowledge toolbox : 11. Impact assessment

KARANJA, Beth
2001

Expand view

Provides a participatory guide that presents a strategy on assessing the impact of resource centre products and services. This guide is meant for resource centres which are reasonably well established and deliver a number of products and services

Toolkits : a practical guide to monitoring, evaluation and impact assessment

GOSLING, Louisa
EDWARDS, Mike
2003

Expand view

This field-tested toolkit has been designed to measure the extent to which programmes make a difference. The 2003 edition of Toolkits has been extended with contributions from SCF and beyond. It describes participatory methodologies, such as mapping and focus groups. It is divided into three sections: underlying principles, practical questions and tools. This new edition brings these up to date and discusses the implications of adopting a human rights approach to development and the increased emphasis on partnership. There are new chapters on impact assessment, monitoring and evaluating advocacy, as well as two new tools - one for improving planning, evaluation, and impact assessment and one for stakeholder analysis

Email forums

LEAP : the learning and evaluation action program

Expand view

The Learning and Evaluation Action Program (LEAP) is an initiative to promote a coordinated and comprehensive approach to learning about the use of knowledge and ICTs in international development. It hosts an e-mail discussion list called LEAP-Impact, a community of practice open to all individuals and organisations interested in the evaluation of information services, projects and products. The website includes posted messages, documents and links to further resources on knowledge management, learning and evaluation for development

Journal articles

Websites

MandE News

DAVIES, Rick
Ed

Expand view

This valuable service includes an electronic discussion forum, numerous papers and resources on evaluation, news and upcoming events, and links to other evaluation resources and practitioners. There is a special section on network evaluation and the use of social analysis in development projects