Responding to complexity in evaluation
Good public policy responds to complexity: it attempts to understand the context into which the policy will be implemented and ultimately, achieve outcomes.
That’s not an easy thing to do. In a recent webinar, I shared three practical tools for designing and evaluating strategic responses to complex issues, and in complex environments namely: systems thinking, outcomes assurance and rubrics.
One of the key points was that we don’t necessarily need to do anything drastically different in terms of the methods we use to collect data for decision-making in complex contexts. Instead, what we need is to think differently. This can mean focusing our attention on a different evaluand (such as the relationship between parts of a system rather than how much impact can be attributed to a program) and paying attention to different indicators of health (for example the core attributes of high-functioning systems Ralph Renger writes about in his Systems Evaluation Theory article). And it means embracing complexity in the data tools we choose – not trying to control the context out.
Systems thinking can create practical and conceptual challenges. It can be less intuitive because of its non-linearity, and considering context in both the design and doing of an evaluation can add layers of complexity.
Another barrier is not being able to see the forest for the trees. While those implementing a systems response always have a much greater depth of insight into the context and their particular sub-system or niche within a sub-system than outsiders, getting a view of the whole ‘system’ from within the system can often be more challenging. Asking questions which challenge status quo thinking can also be difficult for those within the system, due to political and interpersonal sensitivities. Where this is the case, looking outwards for support – from evaluators with an existing systems-thinking practice – makes sense.
In both instances, an evaluation maturity model may be useful. They can help create understanding of the evaluative capacity of those responsible for a response, to identify gaps that need to be addressed, and with setting goals to address them.
Want to understand systems thinking better?
Our most recent ARTD-X Session presented by Kara Scally-Irvine also unpacks benefits and challenges of systems thinking approaches, using the example of a collaborative network looking to reshape a system for greater impact. (You can view that recording here)
You can also read about systems leadership, levers and the importance of changing mental models – in our newsletter foreword here.
