We got inspired by the Impact Evaluation In Practice published by the World Bank in 2016 and we looked specifically at the initial stage of planning an impact assessment.
What we have to develop first is usually our own tailored theory of change. The theory of change paints a picture of where we are right now, where we want to get, and how we are planning to do it. Whether we want to evaluate a project, a company, an institution, or just an activity, we need to have a realistic look at it and to define the present state of it first. We need the analysis of the current state in order to have a so-called baseline for the change, a starting point. This very first step is not necessarily connected to the design of evaluation, however, evaluation strongly depends on the defined baseline, because without it, there would be no way to asses changes.
The change is a journey and at the end of the journey is the desired or wanted state of things. We imagine the state of our project/institution/company/activity after the change and we define this ideal state. What has changed? What hasn’t? Are some processes different, or is it just about the rising number of people taking part in th project? Do we want our clients to be more satisfied and more successfull or is the desired outcome a higher number of organised worshops? It can be both or nothing, it is up to us as the designers to decide, as long as the goals are realistic in terms of financial resources, people and a time-frame. However, the more people involved in the activity take part in defining the aims of change, the more impactful and relevant the change(s) will be.
The theory of change also brings the why and the how, in other words, it explains all that has to happen between the initial and the final state. To develop this, we can create a results chain which establishes the causal logic from the initiation of the program to the end. A results chain will map:
- Inputs– resources of the project (staff and budget)
- Activities– actions/work converting inputs into outputs
- Outputs– tangible products of project activities – directly under the control of the implementing agency
- Outcomes– results likely to be achieved by using project outputs – short to medium term, not directly under the control of the implementing agency
- Final outcomes– final achieved results (Were the goals met?) – can be influenced by multiple factors and are achieved over a longer period of time
The results chain can be used as a basis for formulating the hypothesis that you would test by posing a set of questions. A clear evaluation question needs to be accompanied by indicators that will be measured along the (results) chain. There will be questions and indicators used both to monitor program implementation and to evaluate results. When choosing indicators, it is important to identify indicators all along the results chain, and not just at the level of outcomes. Even if you are only interested in outcomes, it is still important to track implementation indicators, so you can determine whether interventions have been carried out as planned, whether they have reached their intended beneficiaries, and whether they have arrived on time.
Some basic questions to consider ragarding your indicators are:
- Are the indicators (outputs and outcomes) clearly specified?
- What is the source of data for each of the indicators?
- With what frequency will data be collected?
- Who is responsible for collecting the data?
- Who is responsible for analysis and reporting?
- What resources are needed to produce the data?
- What are the risks involved?
As you can see, the first steps in the design of impact evaluations can take a lot of time and effort. However, it is crucial to undertake them well in order to have an evaluation that can trully assess the change, can help us to understand the (un)success of our actions and projects, and which enables us to learn.
————————————————————————————————-
This article is fully based on citations and information taken from the above mentioned Impact Evalution In Practice guide by World Bank (p. 31-43).
