Using Evaluation Theory to Inform Practice
Evaluation is a young discipline, especially when compared to other fields of inquiry like sociology, economics and psychology. Even so, there exists a rich intellectual history and vibrant body of theoretical knowledge that continues to grow and evolve. When I was invited to deliver a workshop on ‘Theories of Evaluation’ for the Australasian Evaluation Society (AES) autumn intensive, there was one take-home message I wanted to convey—if evaluators are not tapping into the 60-year repository of hard-won lessons, they are missing out on considerable wise counsel. This ‘wise counsel’ can greatly improve the quality and utility of evaluation practice.
Without knowledge of evaluation theory, we are susceptible to repeating mistakes of the past and relying on little more than professional folklore and an assortment of methods with no guiding principles for their application.
The formal aims of the workshop were to provide participants with a better understanding of:
- the nature and role of evaluation theory
- major theorists and their contribution
- approaches to classifying evaluation theories
- key ways in which evaluation theorists differ and what this means for practice
- dangers involved in relying too heavily on any one particular theory
- techniques for selecting and combining theories based on situational analysis.
As a passionate advocate of the late Will Shadish’s maxim ‘Evaluation theory is who we are’, I remain committed to the view that evaluation theory is central to our professional identity. Here’s some tough love from Shadish (1997):
If you do not know much about evaluation theory, you are not an evaluator…to be an evaluator, you need to know that knowledge base that makes the field unique. That unique knowledge base is evaluation theory (pp. 6-7).
So what exactly is evaluation theory? A broad answer is that evaluation theory is the body of writings about evaluation that have at their core a concern for practice. Another response is to view evaluation theory as a set of prescriptions about what constitutes ‘good’ evaluation and how it ought to be conducted (as detailed in alternative evaluation models or approaches). Yet another perspective considers evaluation theory as comprising several meta-components including use, knowledge construction, valuing, social programming and practice. In my view, all three framings are important and should be considered within an integrated perspective on evaluation theory.
During the first part of the workshop, we examined these different ways of thinking about evaluation theory, drawing on a conceptualisation that I developed a few years ago to help postgraduate students circumnavigate the ‘theory jungle’. As part of the discussion, we compared the ‘big seven’ theorists: Donald Campbell, Michael Scriven, Robert Stake, Carol Weiss, Joseph Wholey, Peter Rossi and Lee Cronbach, as well as other ‘intellectual heroes’ that have found a place on Alkin’s infamous theory tree.
In the second part of the workshop, I presented an example of how a realist theory of evaluation can be useful for guiding practice, especially when the purpose of the evaluation is to support program improvement and transferability. I also offered some insights from Ernest House about how research on cognitive thinking and bias can prevent evaluators from making fundamental errors when drawing evaluative conclusions. As a group we brainstormed strategies to help determine when and where to use different evaluation theories and approaches, given considerations of time, budget, data, stage of program development, and so on. There is no single or ideal theory of evaluation that will work always and everywhere. It is critical to select and combine approaches in response to situational contingencies.
In the final session, I emphasised the importance of evaluating evaluation theory and considered different criteria that might be useful for reflecting as a discipline on which theories of evaluation are ‘better’ than others. Should we continue to ‘let a thousand flowers bloom’ or is diversity of evaluation models and approaches leading to fragmentation of the field? My own view on this is that we need to get better at determining which theories of evaluation have merit and which are whimsical fads and fashions (or worse, harmful ‘snake oil’). Evaluators need to reflect an evaluative gaze back on evaluation itself.
I invite readers of this blog to reflect deeply on two questions presented to participants during the workshop.
- Who/ what has had the most influence on how you conceptualise and conduct evaluation?
- What guides your current evaluation practice?
As part of the reaction this exercise triggers, I hope that evaluation theory becomes an increasingly prominent resource that you draw upon to inform everyday practice and decision-making.
Thanks to the AES for hosting and organising the autumn professional learning program and to the many participants who attended workshops delivered over this inaugural three-day event.
