Flexibility, Expansive Thinking and Speed
What a year! While many of us are ready to see the back of 2020, there have also been upsides to the year. We’ve seen governments adapt rapidly, communities come together to care for each other, and organisations collaborate to deliver innovative products and services.
It has been a privilege for our teams to work alongside our clients as they have pivoted to achieve their purpose and, through this, support our vision for a more thoughtful world.
Far and away the best prize that life has to offer is the chance to work hard at work worth doing. ~ Roosevelt
In an increasingly complex and interconnected world, evaluation often requires creativity as well as critical thinking. This has enabled us to adapt our approaches and methods to the current circumstances – to capture how programs are changing, the value they are creating in different circumstances, and adaptations that are worth continuing.
Complexity calls for flexibility, expansive thinking and speed.
Certain evaluation approaches have come to the fore in this context.
Developmental evaluation is well suited to understanding innovation in complex systems. It recognises the need for regular review of data to shape the evolution of an initiative and an iterative approach to evaluation. This is vital in novel contexts, where the danger is that hard and fast operating rules and indicators set too early risk irrelevance (as this podcast suggests may have been the case in New York’s schools re-closure at the 3% COVID-19 transmission rate mark).
Systems evaluation is sensitive to our inter-connectivity – recognising that as programs do not exist in isolation, their value cannot be understood in isolation but rather in how they address a need or complement a strength in a system. It’s also particularly useful in the context of a pandemic because it supports rapid information flow.
While there has evidently been a need for some information delivered at speed to inform decision-making during the pandemic, does this mean all evaluation should now be rapid? In our experience, the answer is nuanced. For example, when the Queensland Department of Housing and Public Works pivoted its activity under the Queensland Housing Strategy to respond to COVID–19, our work in partnership with the Institute for Social Science Research at The University of Queensland also shifted dramatically. The Department’s need to quickly understand the effectiveness of its responses meant our work also needed to be sped up. But we can’t reasonably expect that longer-term program outcomes will be evident any earlier than usual, so it’s important to stay the course by measuring program performance and impact over the longer term.
Our collaboration toolkit has also evolved.
Like the rest of the world, we moved our face-to-face processes online. We held:
- interactive online workshops with community organisations to develop their theories of change and evidence collection systems, using virtual white-boards and post its, so they could make the case for the ongoing value of their programs
- online focus groups complemented by Zoom polls and Google Sheets to collect data on what matters and unpack why it matters
- sensemaking sessions using dashboard data and interactive tools, so organisations and the people they support could work together to review their data and what it means for they should do next.
While we’d done much of this before, we hadn’t done it on this scale, week in week out, with as many diverse groups as we now have.
We’ve kept a careful eye on how digital evaluation methods facilitate—or obstruct—accessibility and equitable participation. As restrictions continue to ease and we look forward to a vaccine, I’m keen that we have robust conversations about what mode of engagement is best for the context.
We’ve further unpacked the value of digital.
We’ve found that our experience in evaluating telehealth and digital initiatives has come into new relevance. We’ve worked with phone- and web-based services to understand how they’ve been able to meet shifting needs during the pandemic.
We are particularly excited to contribute to the evidence base indicating the effectiveness of digital mental health initiatives, such as Beyond Blue’s Support Service, knowing how much of a toll COVID–19 has exacted on Australian’s mental health.
With that in mind, I wish you all a restorative break. I look forward to working alongside you in the new year.
You can view more of our thoughts on the year that was in the video below.
Take care
Jade
