Systems Evaluation Theory bookchat with Ralph Renger
Ralph Renger is a leader in the field of systems evaluation, and a household name for evaluators. He’s recently published System Evaluation Theory – A Blueprint for Practitioners Evaluating Complex Interventions (just in time for Christmas gifting to your best evaluator buddy). The book provides a simple three-step framework for applying systems evaluation theory in practice. With plenty of case examples, it’s an accessible read for evaluators at all levels.
We were really excited to catch up with Ralph via video chat – from his home under the blue skies of Tucson, Arizona – to find out more about how he came to Systems Evaluation Theory and what made him write the book.
How did you come to systems thinking?
Ralph originally trained in sports and experimental psychology, where the focus was on cause and effect and randomised control trials, and there was not a lot of room for systems thinking. After several years teaching and conducting evaluations as a Professor in Public Health at the University of Arizona and the University of North Dakota, Ralph started his business, Just Evaluation Services. This gave him more time to spend raising his two girls.
Ralph’s interest in applying systems thinking in evaluation was awakened at a conference in 2011, where he found himself frustrated by the hype around systems thinking, combined with a lack of understanding or deep thinking about how it might be usefully applied in an evaluation context.
“If we couldn’t explain it to ourselves, then how were we going to explain it to non-evaluators?”
This prompted him to write an article, published in the EJA. Not long after, he had a life-changing moment: he was offered $4.3 million and several years in which to evaluate the cardiac arrest response system in North Dakota. The best part? There were no deliverables. It was a huge project, but at the end of it, the new system was saving hours in a context where seconds count:
“At last count, we’ve had over 400 lives saved because of the evaluation.” (Read more about this project and find the article about it here).
For Ralph, this was a clear demonstration of the power a systems evaluation can have, and he has continued to see that impact since.
Alongside juggling soccer coaching, dad responsibilities and book writing, Ralph is keeping busy building a house in Wyoming. Ralph, who had experience in his youth as a building contractor, says a systems perspective really helps when building a home.
“It’s those little things – the wonky stain on the ceiling board that my daughter put on herself – those things create the emergent property of ‘home’,” Ralph tells us, tongue only slightly in cheek. Appropriately, the cover of System Evaluation Theory – A Blueprint for Practitioners Evaluating Complex Interventions, features a builder.
What’s the value of systems evaluation theory and of this book?
Two concepts in particular inform systems theory: interdependence and emergence. Ralph adopts Russ Ackoff’s example of a car to explain both.
“There are several components to a car: the motor, the transmission, the exhaust system, the steering, it goes on. All of those parts, they’re interdependent, meaning they need each other to function: If you don’t have it, it falls apart. So, what is the emergent property of a car? Well, it’s to get you from Point A to point B. No single part can do that by itself: the wheel or the motor by themselves can’t do it. But together they create this property of movement, a car, a form of transportation. That is really important to get your head around, because that transforms how you do your evaluation.”
The failure rate of interventions is high because, as Ralph tells us, they are often delivered as one or two programs that are not sufficient in scope or design, to address the problem. To continue with the car analogy: expecting these one or two programs to get us from A to B is like expecting a steering wheel on its own to perform the function of a car.
Systems evaluation theory provides a way of seeing the system of component parts which create the emergent property, and a methodology that is fit for evaluating complexity and producing actionable recommendations to improve the system.
Ralph has done his fair share of evaluations using traditional approaches: he wasn’t born with an innate sense for when and how to apply systems principles, and he wasn’t always systems oriented. He looks back now at some of his early career projects and can see the value systems thinking would have brought, and the impact it could have created.
“After 19 years or so of doing program evaluation, if 5% of my reports were ever used, I would eat my shirt. And since I began doing systems evaluations, I started developing evaluation metrics to track whether recommendations were adopted, and at the moment it sits around 81%. That tells you that the approach is more meaningful, and it matches the reality in which people are operating better, so that it makes sense to them: if it makes sense to them, they’re going to use it,” Ralph told us.
Our value as evaluators, he says, is to be involved in the design of the program, to bring awareness to the component parts, and whether or not they are working in unison for some higher outcome. Evaluators can also bring a strategic ‘systems’ view to what kinds of other partners are already operating in the system who can be engaged. Using SET, Evaluators can help program designers see these pieces of the puzzle and fit them more strategically together.
Ralph is committed to making systems evaluation more accessible and practical for evaluators to ensure we see the benefits of its value, but he pulls no punches when describing the inherent difficulties we face evaluating complex systems.
“It’s hard work evaluating complex interventions. It takes a lot of time. But if you do it systematically, if you take it piece by piece, you can get through something very complicated and get meaningful results. I just wanted to encourage people that they can do this. This book is for the people who have to DO the evaluation, not just talk about it. The systems evaluation approach, when appropriate, and applied correctly is so powerful for people to make change. I’ve had the good fortune to have that opportunity, and I hope the book conveys that and gives people confidence to take the next step.”
Ralph doesn’t claim the approach provides the only answer, or that it’s best suited to every evaluation. For example, there are some interventions that are simple, straightforward and linear, and where the outcomes are in the direct and immediate control of that intervention to change: the perfect fit for a logic model and a traditional program evaluation approach, no need for a systems perspective!
Nor does Ralph believe that his book should be the end stage of SET’s evolution. In fact, he hopes it is a springboard for more discussion and innovation regarding the application of systems principles, and he is open to evidence-based criticism. For Ralph, it is about providing a platform to implement meaningful improvements:
“That’s the whole point of evaluation, right? Giving people information that they can use to make decisions.”
How to become a better systems evaluator
So how do you become more systems-oriented? Ralph says: take baby steps.
“Just take the time to educate yourself about what systems thinking and systems evaluation is. And just take a baby step. When I was evaluating the cardiac care system, I thought: what’s this thing called feedback loops? Can I find a feedback loop? Oh yeah, there’s one – so how do I know if it’s working? Let’s just evaluate that.”
“So not even looking at interdependence or anything. Just taking a little piece of something that maybe you can get your head around and understand, and just do that piece … try one little system principle and apply it right. You don’t have to get into interdependence or emergence. Just build your confidence up slowly.”
Once we’ve made a start, we can begin to interpret interdependence and emergence. According to Ralph, it is interdependence and emergence that inform the direction of an evaluation:
“If you understand interdependence, it informs exactly how you go ahead with the methodology, and if you understand emergence, it gives you a path forward for evaluating the outcomes. It is important to appreciate that the result of interdependence is emergent outcomes. These are not chronological like a long-term outcome, but outcomes that start to appear.”
This is comparable to the emergent model discussed by Kania and Kramer[1] in their Collective Impact theory, which indicates that solutions reveal themselves iteratively. Progress is incremental and ongoing, and as a result, discoveries are often not a breakthrough, but instead adopt previously unnoticed solutions.
Ralph tends to concur: “The whole is not equal to or greater to than the sum of its parts. It’s the product and that’s something very different.”
You can purchase Ralph Renger’s Systems Evaluation Theory: A Blueprint for Practitioners Evaluating Complex Interventions Operating and Functioning as Systems here.
You can also register for the 2023 bookclub here: https://www.justevaluation.com/book-club
There is no cost for the bookclub other than to purchase a copy of the book.
References
[1] Kania, J., & Kramer, M. (2013). Embracing emergence: How collective impact addresses complexity. Retrieved from: https://ssir.org/articles/entry/social_progress_through_collective_impact
