When is an outcomes evaluation not the solution?
When organisations ask for an outcomes evaluation, the question they are trying to answer is not always “did this program work?” Sometimes the underlying question is much broader: what is the best thing we can do with our money to achieve x?
Generally, this question cannot be answered directly through an outcomes or summative evaluation, unless the scope of the evaluation is to test the comparative effectiveness and efficiency of multiple programs that aim to achieve the same thing. Rather, it is a strategic question.
This is not to say that evaluators cannot support decision makers with strategic questions. We can! But only when you ask us to do so. We call this Strategic Evaluation.
One of the reasons we aren’t explicitly asked to support on strategic questions is that there can be misconceptions about the diversity of approaches available in the evaluation toolkit, what evaluations (or consultants!) can deliver, or a lack of clarity on the purpose of the evaluation. This can lead to requests for an ex-post (summative) evaluation when what’s really needed is strategic advice informed by evidence about solutions to problems. We take a different approach to this than we do to assessing the merit or value of an intervention.
Good news! We evaluators are fairly deft at metamorphosing our solutions to fit the problem at hand (and are used to being a critical friend, so will speak up if we think you are asking for a service that is not going to get you the answers you are looking for!).
How to ensure you’re asking the right questions and engaging evaluators effectively?
To get the best answers, it’s important to get clear on the purpose for which you are engaging an evaluator, and the questions you want answered.
Ask evaluators for strategic advice or strategic evaluation if you have questions like, ‘of the known options, is the thing we are doing the best thing we could be doing?’
An evaluator can support this by undertaking stakeholder consultation to draw together the expertise and experience of a diverse group of people, and by doing desktop research looking at existing evidence about other approaches to solving a particular issue. This can include reviewing evaluations of interventions addressing the same issue in different places.
Generally, the stage of development at which this is helpful is when there are broad ideas about end goals, but there aren’t yet fixed ideas or plans about how you want to get there (for example, if you are at the stage of developing a strategy, but aren’t yet at the stage of having plans to implement it).
Ask evaluators for design and implementation support, or a formative (implementation) evaluation if: you have questions like: ‘what is this project/program/strategy going to look like at a granular level and/or across different implementation sites or with different demographics?’ or, ‘what do the people experiencing this problem want the solution to look like?’
Consulting with stakeholders and the target demographics of the project is often an important step in getting the design and implementation right.
Many evaluators bring strengths in design and implementation. They also offer expertise in things like implementation science, theories of behavioural change and persuasive communication which are used to support more robust design and planning for implementation. An evaluator can help you consider things along the spectrum from the final destination back to what actions and outputs are logically likely to lead to these outcomes, based on best evidence. Program logics are often a part of this work, but customer or user journey mapping, stakeholder or service mapping, prototyping, and desktop research are also often a part of it. This kind of service can also often include co-design or co-production with people with lived experience.
Design and implementation support is useful at the stage at which you have an outline of the solution, but you aren’t sure how it’s going to land with the participants or stakeholders, or in the particular context into which you are delivering. This kind of support can include advice about how to successfully tailor elements of an existing program for a local context or specific demographics, or about how a program is going to work in different operating environments. It may even involve monitoring or evaluating a pilot program!
Ask for a monitoring and evaluation framework to guide your implementation of an evaluation if: you want to answer questions like ‘Is it going to work the way we think it will, and how can we manage the risks of program failure?’
Ask for an outcomes evaluation if: you want to answer questions like ‘is the thing we did the best it could be? Did it do what we thought it would do? And if not, why not and what can we learn from this?’
Still not sure? If you’re not certain about what is needed, the best thing to do is not to pre-define the solution, but focus on clarifying the question you really want answered. Then, speak with a trusted evaluator about the questions you have, and the stage of development your project, program or strategy is at and ask them what service they think you need.
