Navigating Economic Methods of Evaluation – Reflections on Julian King
Many of our clients seek to understand what value for money is being delivered by their programs, but while we frequently see requests for evaluations incorporating a cost-benefit analysis (CBA), this isn’t always possible or necessarily the most appropriate economic evaluation method.
Recently our team participated in Dr Julian King’s Navigating Economic Methods of Evaluation: a practical orientation for evaluators and evaluation users workshop. Julian is a public policy Consultant and Director at Julian King & Associates, with extensive experience in evaluation and value-for-money assessment. Here we unpack some of the key takeaways from a fantastic day of training, with some quotes from those of us who were lucky enough to attend!
Working effectively with economists
We’re fortunate that several of our ARTD team members have a background in economics or experience applying economic thinking to projects. But for those of us who understand economic evaluation but are not trained economists, Julian’s session provided a helpful framework for decision making about when to call in an economist and how to set the relationship up for the success of the project.
“I valued Julian’s emphasis on the importance of high-quality execution by trained practitioners (who also get evaluation). He shared some great tips and language for ways to explain and demystify economic evaluation concepts and methods for commissioners and users of evaluation.”
From our own experience, where an economic evaluation is needed and we don’t have the right training within our project team, it’s important to bring our economist partners in early, and involving them in the evaluation’s design, as this helps to shape KEQs, indicators and data collection tools that will ensure the right data is collected to make economic assessments possible!
The session with Julian also provided a concise overview of all the different economic evaluation approaches that are possible – from cost-benefit analysis (CBA), cost-effectiveness analysis (CEA), cost-utility analysis (CUA), break-even analysis, or social return on investment (SROI), and how to select the right approach to fit the project’s purpose and context. This provided a great basis of shared concepts and language that will help us with clear communication when working with our economic partners, and also strengthened the confidence of our team in communicating with clients about when an economic approach may not be fit for purpose.
“I came into the session with very little experience in Value for Investment, and left with a much clearer understanding of the different approaches available. Rather than feeling apprehensive when I see VFM or cost-benefit analysis in an RFQ, I now feel more confident asking: is this actually possible, and if not, what could we offer that would give the client the evidence they need?”
The number is not the judgement
Those of you who have kept abreast of Julian’s developing approach to value for money assessments may already be aware of his Value for Investment (VFI) approach. After stepping us through the different economic evaluation approaches, Julian provided a tour of VFI. This uses the steps of the ‘logic of evaluation’ (Scriven 1980; Gullickson 2020), combined with 4 principles, to place economic methods within the broader context of the evaluation (you can view the graphic on Julian’s website here, and more on the logic of evaluation here). This resonated strongly with how we think about integrating economic approaches – or any approaches, really – within an evaluation. Julian’s VFI framework really emphasizes that strong evaluative reasoning, combining explicit criteria and standards with interdisciplinary and mixed methods approaches – as well as engaging stakeholders in questions of what constitutes ‘value’ – are essential ingredients for a sound evaluative judgement. It is these elements that support reasoning about what costs and benefits get included, and how they are valued. And making these workings and assumptions transparent is critical to creating convincing evaluative arguments.
The VfI framework makes it quite clear that while the evaluative judgement may be informed by a number (the magical dollar figure that represents the quantifiable benefits vs costs, or similar), the number is not the judgement, nor is it the whole evaluation (see the excellent blog Ten Commandements for Social CBA and SROI for more).
Using the VfI framework to inform evaluations with an economic component helps to focus attention on some really important questions:
- what is the value proposition of the investment (program/policy/strategy)?
- what are our criteria or standards for success? (What does good look like for economy, efficiency, equity and cost-effectiveness)
- what evidence do we need to determine if what is being achieved is good?
- what quantitative, qualitative and economic methods are the best ways to collect and analyse that evidence?
- from the evidence, what conclusions can we draw? Is it good? Compared to what? And for whom?
- is it good enough, considering the investment? How could more value be created?
Building confidence in Value for Investment
Overall, the workshop allowed those of us with less formal education in economics to build our confidence in the concepts, language and applying strong reasoning to the selection of economic methods. And for those of us with more experience in economic evaluation, it was a chance to pause and reflect on practice, and consider what else we can incorporate to continue building rigour into economic components of evaluations, as well as to strengthen how we communicate with evaluation stakeholders about economic evaluation methods and findings.
“The session reinforced that economic evaluation isn’t simply about applying a method or arriving at a number. There are important questions and tensions to work through- around attribution, assumptions, data limitations and what we can confidently conclude. Having a framework and shared language for navigating those questions is particularly useful when working alongside economists and evaluation commissioners.”
It was great to have Julian on hand to help us puzzle through a few applied examples from our work, and answer a plethora of questions!
We’d like to thank Julian for his time sharing his expertise with our team. His workshop helped to solidify concepts in a way that was practical and made sense to the work we’re doing.
If you’d like to find out more about the workshop, you can do so on Julian’s website here: Julian King & Associates Limited | Consulting Services
Julian also writes a Substack where he shares thinking and insights on economic evaluations, value for money and related topics. It’s a useful resource for anyone wanting to dig a little deeper into some of his great thinking, and the ideas covered in the session.
