Learning and applying evaluation theory – the highs and lows in the first year doing evaluation
It is not always necessary to have a thorough knowledge of every theory in a particular field in order to be able to work in it. But I have often found that getting to grips with at least some theory – the underpinning assumptions, frameworks created, limitations that exist as well as general gaps in the literature – helps me to practically contextualise and navigate the real world. It’s not always easy, however, especially when you’re just starting out in a field like evaluation. Here is why I have found it useful to learn evaluation theory, my discoveries about what helps me to learn, and some of the challenges I’ve encountered in my first year as an evaluator.
Why is it helpful to learn evaluation theory?
There are many different ways to approach evaluation and getting comfortable with the knowledge that there is no ‘silver bullet’ approach can give you more confidence to make decisions that are best suited for that particular evaluation.
Having knowledge of a range of theories (and theorists) to draw on can help us more skilfully craft evaluations that are adapted to different evaluation contexts and need. Developing a foundational knowledge of evaluation theories and concepts provides you with a range of tools and options that act as a starting place to help guide decision making. This is particularly useful in the evaluation and tool design phase.
We are never working with a blank slate! Understanding evaluation theory also helps us to better understand the ideas, learnings and assumptions that may be driving the decision making of evaluation commissioners and collaborators.
Evaluation in practice is often iterative – the more you learn as you go, the more improvements you can make to tools and approaches. Theory can also help to guide these decisions, so they are based on a strong rationale, rather than simply reactive.
Challenges
Theories are often designed with an ideal state in mind, meaning they are removed from the day to day context that we operate under as practitioners. Real world evaluation contexts don’t always allow for the delivery of an evaluation in the way prescribed by the theorist. There are lots of contextual factors that get in the way: budgets and time constraints, the questions you need to answer, the available data, challenges with recruiting participants, and a myriad of other factors. We must do our best in the conditions we’re operating in to create evaluations which are rigorous and valid – but which also enable us to provide quality and timely advice to help our clients make decisions. Often to determine ‘what is reasonable’ in terms of validity and adapting theory to real world context uses judgement, which is something you develop over time and with practice – so this can be hugely challenging when you’re early in your career.
There are some other challenges that can make learning about evaluation theory feel like climbing an epic mountain:
- Evaluation draws on theories and knowledge from many different disciplines. This is one of its strengths, but when learning, can make it hard to grasp distinctions between what’s evaluation theory, and what’s research, or public administration, or change management!
- As in most fields, there is a lack of consensus in the evaluation community around methodologies, how and why things are done a particular way, and even over what constitutes ‘knowledge’. So again, when learning, it’s difficult to know which viewpoint is right for your context – and again this requires judgement!
- Theories are constantly evolving! So just when you think you’ve figured something out, you may soon find you’re out of date.
- The Dunning-Kruger effect is alive and well in the field of evaluation – the less you know, the more you overestimate your abilities; the more you learn, the more you know you don’t know and that does not help with confident decision making!
There’s so much to know in the world of evaluation! As evaluation is constantly evolving, so must we as evaluators be – as a result, the overwhelm is often real!
Image source: https://medium.com/geekculture/dunning-kruger-effect-and-journey-of-a-software-engineer-a35f2ff18f1a
What helps me learn?
Luckily, there are also some things that do make learning about evaluation much easier. Here are some of things I have found helped me this year.
- Reading about frameworks for decision making and application of theory in practice. I am loving the work by E. Jane Davidson, who brings a lot of her real life experience into how she talks about evaluation theory.
- Hearing other evaluators talk through why they made this or that decision (i.e. the steps they take in making a judgement), as well as about the challenges they experience helps me to make sense of my own experience! It’s important as an evaluator to know you’re not alone in experiencing uncertainty – that everyone is questioning everything all the time! It gives you back some of the confidence that the Dunning Kruger effect has ripped out of your hands – because it means uncertainty is part of the normal process of doing evaluation!
- Trying to explain theory to other people and putting these ideas into my own words is so helpful – as is discussing how they would be applied in practical situations to test my knowledge. I’ve had the opportunity to discuss what I’ve learned about value for money assessment using rubrics, for example, to assess whether and how it would apply in a particular evaluation design as we responded to a request for an evaluation.
Takeaways
Given hearing from other evaluators about their challenges and things that don’t go right is helps us all to learn – please feel free to share your own challenges (current or past) in learning about evaluation theory, and what helps you learn, over on LinkedIn!
