Program vs Systems Evaluation: Cardiac Care Case Study Article
Partner Andrew Hawkins has co-authored a new journal article in The Canadian Journal of Program Evaluation on the application of systems thinking concepts to evaluate systems. The article uses a case illustration of the Cardiac Care System in the US to help teach practitioners the difference between a systems vs program approach to evaluation, and how to apply a systems thinking approach.
What’s the difference between systems evaluation and program evaluation?
Ever wondered what would happen if you had sudden cardiac arrest in a remote location in the mountain west states of the USA? Very likely, you’d hope there was a system around you to get you to a hospital, save your life and provide sufficient care to allow you to walk out of hospital pretty quickly. That’s no mean feat.
We expect a lot from the interconnected and often invisible systems that surround us every day: more than can be provided by specific programs. Systems thinking in evaluation requires a switch from thinking of public policy in terms of welfare ‘programs’ or ‘interventions’ to the overall set of relations that determine a particular outcome. Program evaluation tends to be focused on the value of a specific intervention. Systems evaluation is focused on understanding the value of an intervention into a specific system. This means systems evaluation seeks to understand the value of a program or intervention not on its own merits, but how it interfaces with an already existing system.
The list of systems we encounter is endless and overlapping: the traffic system, the health care system, the education system as well as the welfare system and all the other messy and complex social systems into which an intervention is placed. This can make systems evaluation sound overwhelming and the comfort of a less ambiguous program evaluation more appealing. This is one reason why any systems evaluation has to start with defining the boundaries of the system for evaluation. Defining the system is a pragmatic concern more than anything else. It is often based on who is willing to engage rather than any deep philosophical position about where a system starts and ends. In this way systems evaluation tends to merge with strategy and management and is by its nature concerned with collective impact, place-based interventions and whole of government solutions. We think it has a lot to offer in addressing wicked problems.
After a road trip through Montana and Wyoming my friend and colleague Ralph Renger and I discussed the emergency cardiac care system. We’d been on the road to observe the cardiac care emergency response 
system; from the emergency dispatch centres that take 911 calls, to the paramedics, and various hospital staff involved in patent care (including the introduction of a mechanical CPR device that occasioned the perceived need for an evaluation).
Over a meal in Main Street Cafe in Buffalo, Wyoming, we discussed the value of a systems thinking approach to capture a better picture of what was affecting cardiac patient outcomes, such as hospital protocols around accepting patients, including correct use of the CPR machine. This led to a discussion of the similarities and differences between program evaluation and systems evaluation and the need for a pedagogical paper.
So, what does it mean to evaluate a system?
Ralph’s big point was that in a systems evaluation, the system is the unit of analysis, not any one program. This is a cognitive shift for many program evaluators, given the usual focus on programs and outcomes. While all social programs intervene in a complex social system, much valuable public policy exists in a system free of, or at least not defined by, its ‘programs’. So, any intervention into a complex (and adaptive) system will only be effective to the extent that it addresses a need or complements a strength in that system. A systems evaluation mightlook at a purposeful set of actions performed by a group of actors, rather than limiting the view to a ‘program’, to get a picture of the interactions that may hinder, change or contribute to an intervention’s intended outcome. A systems evaluation is less concerned with what works in general, and more with what works here and now. It focuses on rapid information flows and problem solving.
This paper seeks to provide practical insight into the value of a systems approach to evaluation using the relatively simple case study of a rural cardiac care system. We hope it is useful to those working to plan and achieve better outcomes from social policy.
Read the article: ‘Comparing and Contrasting a Program versus System Approach to Evaluation: The Example of a Cardiac Care System’ in the Canadian Journal of Program Evaluation: https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/68127
Learn more about Systems Evaluation: Ralph Renger is presenting an AES Webinar on Systems Evaluation Theory with Lewis Atkinson and Brian Keogh, 10 December 2020. Book online.
