Building the bigger picture: using the SDGs in evaluation
When evaluating, should we be focusing solely on the program or should we endeavour to consider its entire ecosystem?
At the Australasian Evaluation Society NSW event last month, Michael Reid, Managing Director at The Keyline Group, looked at how we can use the United Nation’s Sustainable Development Goals (the ‘SDGs’) as a framework for taking a ‘systems thinking’ approach to evaluation.
So, what is systems thinking?
Systems thinking recognises that nothing occurs in isolation. We live in a complex and interconnected world, with many interrelated actors and parts. While there is not universal consensus on how to define systems thinking in evaluation, a key underlying concept is that the parts that make up systems are dynamically intertwined and won’t be wholly understood if considered in isolation.[1]
Evaluating an intervention using a siloed approach may mean we miss factors in the surrounding environment that contribute to successes or failures. Take a hospital’s emergency department, for example. Evaluating an emergency department might tell us that 90 per cent of people are waiting less than four hours, which exceeds the KPI set by the hospital. But, without conducting an evaluation of the hospital’s entire system, we may never be aware that achieving this target drew hospital personnel away from the wards and a significant decline in client satisfaction in other areas of the hospital.
Evaluators using systems thinking want to be able to understand the relationships within a system because these can provide the best insight into: (a) what is happening; (b) why it is happening; and (c) how progress can be achieved. To do this, Reid says that we firstly need to define the system itself, and then ask who is at the centre. If we put individuals or community in the middle of a system and then examine the interrelationships between the actors and parts surrounding them, Reid argues that we will get the best understanding of what works and why.
And where do the SDGs come in?
The United Nation’s Sustainable Development Goals (SDGs) were unanimously adopted in 2015 by all countries as a call for action to promote prosperity while protecting the planet. The SDGs comprise 17 goals, 169 targets and 230 indicators; Reid describes them as a ‘global map’ of a system that all countries can use for evaluation.
While providing a streamlined model all countries can apply, countries can tailor the indicators and targets to their specific circumstances. The SDG Transforming Australia project collects evidence about Australia’s progress towards achieving the SDGs, focusing on national issues and problems that we face. The goals, targets and indicators can provide a lens for guiding national evaluations, to better understand the impact of an intervention on the broader system, or vice versa.
The International Institute for Environment and Development has provided five considerations to guide national evaluation agendas using the SDGs. These combine the learnings from the Millenium Development Goals evaluations and use a complex systems perspective.
- Think beyond individual policies, programs and projects by examining issues that cut across different sectors.
- Examine macro forces influencing successes and failure by considering political, economic, ideological, environmental, socio-cultural and technological circumstances.
- Take into account multiple definitions and measures of ‘success’.
- Recognise the importance of culture and its influence on societal behaviour.
- Shift towards evaluative thinking and adaptive management to recognise flexible approaches to governance and management.[2]
Beyond using the SDGs to frame national evaluations, Reid proposes that we can use the SDGs to evaluate individual programs at both a state and agency level. He thinks this approach will give us the best understanding of how to manage the impacts of a program, whether they be positive or negative. Authentic partnerships between business, government and the community will also provide the best results.
One project that has recently been evaluated in this way is the NSW EPA Organics Waste Programs. The evaluation attempted to map the EPA’s waste programs to each SDG to obtain a comprehensive understanding of their impact and understand any unintended consequences. To provide an example, they identified that aiming to improve the gender makeup in leadership at the EPA may not only impact Goal 5 ‘Gender Equality’ of the SDGs, but also waste management. This is because a greater gender balance in management positions could mean better decision-making processes.
But how feasible is evaluating an entire system for one program?
Reid’s presentation sparked lively discussion among AES members and several people raised issues they thought would come up when using systems thinking at a micro level.
While it’s one thing to use the SDGs to think about impact holistically, assessing causation and impact is a more complex question. Using the example above, how can we be sure that having more females in leadership positions leads to better decision-making processes and, in turn, waste outcomes? There could be a range of factors that are contributing to this.
Relatedly, how do we define and put boundaries around a system? If we were trying to evaluate a program aiming to reduce obesity in young people, how wide would we need to stretch the systems lens?
At ARTD, we see real value in thinking about the impact of a program or intervention on the broader system and looking at unintended consequences that may arise. In fact, Jade Maloney and Katherine Rich are talking about how you can build systems thinking and address systemic barriers through program design at the AES conference in Launceston on 17–21 September. We look forward to more discussion and unravelling of some of these questions and challenges further.
