Ancient wisdom and systems evaluation
As I sit down to reflect on this Mabo day I am listening to my most played Spotify song of 2019 – My Island Home by the Warumpi band. Spotify serves up an eclectic mix of songs that provides me with a window into my heart and reason to contemplate my soul. It gives me great joy to know my best friend, an Australian tour guide, now plays this song to his American tourists every time they make the trip from Cairns to Alice Springs. This song of longing for home clearly resonates – the bittersweet remembering of a time long ago is a sense that I think most of us WEIRD (Western, Educated, Industrialised, Rich, Democratic) people share, somewhere in our heart, when we are in the right time and place. For me, much of the time this memory is buried beneath a dizzying array of modern world responsibilities and a frantic pace of daily life. But I can’t shake a constant feeling of forgetting something important from the past, like the pervasive background radiation remaining from the big bang. Often and unfortunately the call on my attention is no stronger than the unsettling feeling of having left something important on the kitchen bench. Yet I am deeply concerned that our leaders are no better than me at accessing ancient wisdom and bringing it to the forefront of our decision making to address the problems that beset our society. It’s a gnawing feeling that you can’t solve a problem at the same level of consciousness at which you created it, that has fuelled my interest in ancient wisdom, both western and shamanic since I was a teenager.
Tyson Yunkaporta speaks to me like few other authors. Sand Talk[1] is the book I have been waiting to read for at least 20 years. Tyson’s ability to articulate a way forward out of this mess that focuses on the ‘how’ of Indigenous ways of knowing rather than that ‘what’ is spellbinding. Tyson moves between simple metaphor, complicated narrative, and complex solutions in way that puts tired old ‘evidence-based policy’ to bed. I don’t always agree with everything he says, my western logical brain cannot fully give up on a preference for watertight argumentation, my Nietzschean tendencies mean I am inclined to smash the idols of popular opinion, but his journey is one of unbridled curiosity that I find compelling. I feel a sense of connection to Tyson as a fellow traveller, attempting to ride the waves of ancient wisdom to make sense of the here and now. Tyson’s work is far too deep for me to make sense of on one reading – and maybe it’s the very desire to make sense that is holding me back. I get the sense that Tyson too is grappling with the knowledge that he has received, his enthusiasm and admissions of his own shortcomings are familiar as is his simultaneous impulse to subject this wisdom to critical reflection. As I try to incorporate his teachings into my own knowledge, I am immediately drawn to the similarities in how he describes the world and Indigenous ways of knowing with the concepts of systems in evaluation.
There are many approaches to systems thinking. All those I have encountered seem to involve consideration of multiple interrelated or interdependent parts forming a connected whole. A system might be natural such as a wetland, an ecosystem full of fish, birds, frogs and river flows, or it might be man-made, such as the environmental and resource management policy system. This man-made system may be comprised of farmers, public servants from multiple agencies and jurisdictions, environmentalists, politicians, lobbyists, media and others whose opinions and actions are interacting with the fish, birds, frogs and river flows, as well as with each other. While not all the agents in this system have a voice, this kind of system or ‘networked governance’ will be very familiar to anyone involved with public policy.
Evaluation as applied to public policy is generally concerned with the question of the value of certain polices, programs or interventions. It is implicit and taken for granted that these interventions are made into an existing context or system. Many approaches to evaluation seek to control for the effects of context to allow us to reach some judgment about the independent impact of a program. One such approach is the randomised controlled trial (RCT). But systems evaluation sees the success of a policy, program or intervention as tied to the context in which it is delivered. To a systems evaluator, there is little point controlling for context, or asking what the value of an intervention or decision until the system itself is understood. On this account change in a system is always due to the effects of innumerable factors, often interacting in complex ways that cannot be understood in isolation or by controlling for context.
Systems evaluation is useful in those situations where it is very difficult, if not impossible, or simply not useful, to attribute the changes in a complex system to any one intervention. We have all read reports that observe some change, such as the quality of a wetland, and then struggle with attributing that change to one or more interventions. It is often practically impossible to untangle the web and control for ‘context’ with an experiment or even quasi-experiment. Many evaluation reports lament this lack of control. The desire for experimental or statistical control as a path to understanding lies deep in the western scientific psyche. We often aim to break down, catergorise and ‘solve for x’. This is the core of the scientific method, of Taylorism and scientific management. This approach has led to wondrous advances in medicine and amazing efficiencies in the output of modern factories. However, applicability to complex social systems has been in question for decades. It is obvious in most human endeavours that we can learn from the past, but we know we cannot rely solely on the past for decision making in an uncertain future – to do so is to be the proverbial general who is always planning for the last war. Yet social policy is littered with examples of attempts to reduce decision making about the value of interventions to the results of past endeavours.
General Stanley McChrystal may seem to live in a world far away from that of Tyson Yunkaporta both figuratively and literally. As an American military General in Iraq one might cast him as the epitome of the western world desire to control and expropriate resources as efficiently as possible. McChrystal was a product of the discipline of West Point academy and the American military tradition – I cannot picture the curious Tyson Yunkaporta working in this environment. Confronted with the complex problem of fighting al-Qaeda in Iraq McChrystal became highly critical of the tradition of scientific management and ‘the one best way’ associated with Taylorism and the search for efficiency.[2] Concerned with real-world outcomes in complex environments, he moved away from an instinct for ever more efficient planning and in 2003 took on an explicitly complex systems approach in reconfiguring the Joint Special Operations Task Force in Iraq, a group of elite soldiers from across the Army, Airforce and Navy. His approach for generating outcomes was to focus on emergence and learning. He says ‘Our entire force needed to share a fundamental, holistic understanding of the operating environment and of our own organisation, and we also needed to preserve each team’s distinct skill sets. We dubbed this goal—the state of emergent, adaptive organizational intelligence—shared consciousness (italics in original), and it became the cornerstone of our transformation’. Emergence of new phenomena—where the whole is more than the sum of its parts—is a key systems concept. Non-linear outcomes is another. This is when change happens suddenly despite a constant flow of inputs, like a river that for many months is resilient to increased fertiliser run-off, and then suddenly when the fish cannot eat sufficient amount of the increasing algae a tipping point is crossed, the algae explode in number, and the fish suffocate. Up until this point the fish were thriving on the additional algae—until they weren’t! Monitoring fish stocks by assuming linear growth would be very deceptive. Both emergence and non-linear outcomes make it hard to study phenomena operating in systems through a narrow focus on the value of individual component parts—whether that be a river system or an elite military operation.
A system may be thought of as simple, complicated or complex, but the distinctions are ultimately of degree not kind. A simple system may have relatively few parts whose individual function is easy to understand, such as a paper aeroplane propelled by an elastic band. A more complicated system such as a jet engine will have more parts and each parts’ contribution to an outcome might be harder to understand. A break in one part will still have flow-on effects but across a large scope of activity, such as when a jet engine fails, When the parts are not only interdependent as in an engine (i.e. one part breaks and the engine stops) but adaptive (e.g. the pilot makes a decision as a result of a warning light and a few other dials, that in turn leads to another set of indicators and more decisions) the emergent outcome may be non-linear and much harder to predict. Much has been written on types of complex adaptive system, including by the Santa Fe institute that goes far beyond the scope of this article.[3] Suffice it to say that the presence of a high degree of adaptation by agents in a system changes everything. If the set of indicators and dials on a plane are complicated, and decision making by an airplane crew is complex as well as adaptive, then a complex adaptive system is what happens when the buttons are moving around and do different things each time you press them. And people are complex and adaptive.
The greater the number, diversity and interdependencies between factors generating outcomes relative to the intervention, the more useful a systems approach is likely to be for understanding how to make positive change. For example, a water management plan relies on the cooperation of many levels of government and different actors as well as changes in the natural ecosystem, where run-off is just one of a multitude of factors. Evaluation may not be able to ‘control’ for the lack of rain (and all other factors affecting outcomes) to measure the impact of the plan on water levels and river or wetland health. So rather than seeking to measure how much the plan changed the wetland, a system evaluation may seek to understand the weak points in the policy systems designed to support river health, and then respond to that. This makes systems evaluation intimately connected to the public policy making process, concerned as it is with what is likely to work ‘here and now’, not with what worked ‘there and then’.
Taking a systems approach does not mean we should abandon experiments or RCTs, just that these experiments are likely to be most useful when applied to relatively small scale components in a system. The scientific method is about isolating and testing individual mechanisms – the method was never intended to be adequate for understanding social programs comprising multiple mechanisms operating in complex social systems. If we move from seeing interventions as having an innate and unvarying value, and instead consider the value of an intervention as tied to how it interacts with other factors in a specific context, then we can measure change in one context, make reasonable conclusions about the causal factors using an appropriate method of causal inference[4], and drop the question of whether these same changes would happen in another context. This makes evaluation less ambitious but more useful to the problem at hand. The dynamics within a system and our limited ability to understand means that monitoring and evaluation becomes a continuous process of asking ‘does this approach make sense?’ and ‘what evidence do we have for that?’ and does not lend itself to finding ‘the answer’ and then implementing it. It may mean surfing the constant flow of information between the parts or ‘stocks’ of information that make up a system as defined by Donella Meadows[5].
Considering public policy and its evaluation it may be useful to distinguish between systems that are designed for a specific purpose, such as a coordinated emergency response, and those arise naturally without any man-made purpose, such as homelessness or river flows and those that sit somewhere in-between, like the social housing system that has multiple contested purposes. The work of Ralph Renger[6] has focused on planned and complicated systems. I was lucky enough to observe Ralph working on his system evaluation of remote cardiac care in the mountain west states of the USA. The focus in this kind of systems evaluation is on understanding the important players and dynamics within a system. The tricky first step is defining the system and its purpose. While an ecosystem may seem to have some obvious geological boundaries, in a purposive public policy system it may be defined using pragmatic considerations. It may be defined based on those with the power and interest in making a difference. Evaluation that applies this kind of systems thinking is often involved with mapping the system and using data and evidence to improve the efficiency of the system, and sometimes to change it in fundamental ways.
Other systems lack a clear and common purpose. Social housing involves many different people experiencing varying degrees of housing instability and a wide array of other needs interacting with a broader policy context that is at times supportive, and others coercive. Homelessness itself is not a system that anyone designed. These systems may be constantly changing, both in terms of inputs and who is involved, as well as their relations with one another and their individual aspirations, knowledge, attitudes and behaviour and the broader economic context. This is the world of complex adaptive systems as described in the Cynefin framework by Kurtz and Snodwen[7]. This framework warns us against the old ways of categorising programs and working out ‘what works’ or identifying ‘best practice’. A complex adaptive system often has so many interdependent parts that are moving faster than our ability to understand. Here the ability to find enduring cause and effect relationships may not be possible – we may be better monitoring changing patterns and making small scale and responses to try and nudge a system towards a more desired state. The goal here is adaptation and effectiveness – efficiency is not really feasible because there is no ‘one right way’ that can be discovered and implemented.
As should be clear by now, a common thread in different approaches to systems evaluation is doing away with the idea that you can usefully control for context. It shares this orientation with realist evaluation. Controlling for context with experimental design is an unfortunately 19th century view of science, using 20th century techniques of experimental control but applied to VUCA (volatile, unpredictable, complex and ambiguous) 21st century problems. The nature of these problems stem from a vastly more interconnected and interdependent world where the behaviour of one person or institution can lead to rapid and unpredictable outcomes across the system. This is now a commonplace idea – but not one that has found itself into the inner workings of social policy or its evaluation.
If the failures of economists to predict economic downturns, or the widespread inability to replicate the results of RCTs applied to social programs can teach us anything, it is that evaluation should be an exercise in humility. We should be very careful what we can infer from any particular study that has any lasting relevance. Evaluation and evaluative thinking are crucial, but specific evidence often has a very short shelf life. We should focus on more regular monitoring, evaluation, and the use of evidence in decision making, and less on plans to solve the problem of ‘what works’ in any particular social policy field. As Tyson Yunkaporta says, ‘adaptation is the most important protocol of an agent in a sustainable system’. We should be realistic about what our interventions may be sufficient for achieving and we should not anticipate that what worked here and now will work there and then. We need to unburden ourselves from our mental model of a clockwork universe, where programs either work or don’t work and are easily replicated and scaled up. Programs are not complicated machines. They are living, breathing organisms that adapt and thrive in some contexts and become extinct in others. Tyson reminds us of the wisdom of his elders ‘if you don’t move with the land, the land will move you’.
The western reductionist mindset and scientific management have got our western culture this far. Indigenous wisdom has always been there, waiting. Systems concepts may provide a bridge. When Tyson Yunkaporta says ‘Sustainability agents have a few simple operating guidelines or network protocols, or rules if you like – connect diversity, interact and adapt’ and McChrystal agrees ‘the difference between command and control on the one hand and adapt and collaborate on the other was the difference between success and failure’ there is much room for our two ways to work together in addressing problems that exist in complex systems. On this understanding, respect for ancient cultures becomes more than ‘the right thing to do’ it becomes an ‘essential thing to do’. There is much opportunity for us to share and learn from one another about the place of humanity in our environment and the value of the interventions we make, if only we would slow down, listen and think.
What Aboriginal people ask is that the modern world now makes the sacrifices necessary to give us a real future. To relax its grip on us. To let us breathe, to let us be free of the determined control exerted on us to make us like you. And you should take that a step further and recognise us for who we are, and not who you want us to be. Let us be who we are – Aboriginal people in a modern world – and be proud of us. Acknowledge that we have survived the worst that the past had thrown at us, and we are here with our songs, our ceremonies, our land, our language and our people – our full identity. What a gift this is that we can give you, if you choose to accept us in a meaningful way. -Galarrwuy Yunupingu – The Monthly July 2016.
Image chosen/ added by Holly Kovac,
ARTD’s Aboriginal Research Assistant
References
[1] Tyson Yunkaporta. (2019). Sand Talk: How Indigenous Thinking Can Save the World. Text Publishing, Melbourne Australia.
[2] McChrystal, S., Silverman, D., Collins, T., & Fussell, C. (2015). Team of Teams: New Rules of Engagement for a Complex World. Penguin.
[3] https://www.santafe.edu/research/results/papers/1383-complex-adaptive-systems
[4] See for example, Rogers, P, Hawkins, A. J., McDonald, B, MacFarlan, A, & Milne, C. (2015). Choosing appropriate designs and methods for impact evaluation. Australian Government Department of Indusry, Innocation and Science.
[5] Donella Meadows. (2008). Thinking in Systems: A Primer. Chelsea Green Publishing.
[6] Renger, R. (2015). System evaluation theory (SET): A practical framework for evaluators to meet the challenges of system evaluation. Evaluation Journal of Australasia, 15(4), 16–28.
[7] Kurtz, C. F., & Snowden, D. J. (2003). The new dynamics of strategy: Sense-making in a complex and complicated world. IBM Systems Journal, 42(3), 462–483.
