Evaluative thinking under pressure
Over the past few months, we’ve observed how hard it is to make decisions under the cloud of uncertainty. As evaluators, we’re used to exploring context and drawing on the available data to make good decisions, faster. But we’ve realised that, in a crisis or other rapid change, a traditional approach to evaluation isn’t practical.
We’ve developed The 3R Framework to support decision-making under pressure. It’s based on the principles of action research and developmental evaluation and is a way of considering how we react in a crisis situation, and how evaluative thinking might provide a way to navigate through it with purpose.
The essence of the framework is:
- Using ‘good enough’ data that you’ve already got
- Maximising learning opportunities available to you, right now
- Drawing on instinct and heuristics that are helpful and beneficial to you and your stakeholders
- Raising awareness of human factors that might be counterproductive or dysfunctional, so you can avoid them
Our framework is based around questions because, in our experience, being an evaluator is about asking great questions and not about having all the answers!
You can use the framework to direct responses at an organisational, team, program or individual level. If you’re applying it within your organisation, team or to a program, we suggest getting a diverse group of people together to reflect, drawing on existing data and creating new data from the stories and experiences they share.
You can watch Sharon Marra-Brown describe the Framework in this short video.
Earlier this week, in the first of our Business as Unusual webinar series, we discussed the framework with colleagues from across Australia and the world. With their permission, we’d like to offer a summary of how evaluators, government agencies and not for profits are reacting, reflecting and refining their approach to the current Covid-19 pandemic.
REACT
Your initial reactions to a crisis—whatever they are—are normal, natural and necessary. We talked about how huge and overwhelming the response to Covid-19 has been for them, their families and their organisations. Most people felt their organisations had reacted well, having to be flexible, agile and creative to keep meeting their client’s needs.
Reactions can be positive, or they can be negative. For example, pausing all activity can be a positive (giving time to slow down and think) or it can be negative (unnecessary delays with consequences for service provision). Decisions and actions in this phase can lead to unintended consequences. It was great to hear that some of the reactions have led to innovations and improvements. For example, meetings online can take longer to prepare for and execute, but this can lead to greater clarity of thought and more considered actions.
REFLECT
Many people seem to be entering a reflective phase. The role of evaluation during the reflective phase was contested: some colleagues are turning to evaluators to explore how the ‘new normal’ for program delivery is working. Others are pausing evaluations to allow program staff to focus on delivering.
We discussed a trend towards people being more engaged and thoughtful about how and why we collect data, and more transparent in communication about the use of data. Together, we discussed how we can deliberately create opportunities for structured reflection—many of us are ‘reflecting on the run’ due to time pressure.
It’s clear that these pressures require us to rely more on ‘good enough’ data to make decisions. Perfection isn’t practical right now and, while we need to be mindful of data limitations, we can’t be paralysed by them. We discussed how important it is to seek out diversity: bringing different people, with different data (including anecdotal evidence) will give better information for decision-making that relying on a single data source, such as a survey or administrative data, that does not cover everyone.
One of the suggested additions to our framework was a question around the assumptions we make as service providers and evaluators. For example, one person working with remote Indigenous communities assumed that access to technology would be an issue for participants. But it wasn’t: shifting to online has worked well. We also want to ensure the framework guides us to consider opportunities. One participant said Covid-19 has helped them re-think program design: they were forced to drop what they thought were ‘essential’ program components, and in doing so learnt they were ‘nice to haves’ not critical for achieving outcomes.
REFINE
Moving towards refinement (which many felt aligns with loosening Covid-19 social distancing restrictions) we need to be mindful that not everyone is ready to reconnect. There is a lot of grief associated with this pandemic: we have partners who have lost jobs, funerals we couldn’t attend and people whose mental health is suffering. We can’t just flick a switch and go back to where we were two months ago. We discussed how people’s capacity to accommodate and respond to change will be different as we go forward: some may be less able to adapt; others will be keen to change. Communicating clearly will be crucial.
Looking forward, some participants noted they will be working alongside evaluators to anticipate future data needs and making changes to their monitoring systems now.
OVERALL
It’s possible—particularly if you’re responsible for many programs or a large team—that these phases will be happening all at once. Also, reaching a feeling of ‘refinement’ is not an endpoint: as new information and scenarios unfold, we may find ourselves stepping back into more reactive decision-making.
It is clear that strong, compassionate and flexible leadership will be fundamental to our response. Our decisions must be the best ones we can make at the time, using the best data we have available. The 3R Framework is a way of ensuring evaluative thinking, even as the world around us shifts quickly.
If you would like to use the 3R Framework to guide your thinking, here is a downloadable graphic, including questions for each phase.
Our next Business as Unusual webinar is Sensemaking Under Pressure. In this interactive online workshop, Jade Maloney will be joined by a team of experienced facilitators to help you apply Chris Corrigan’s ‘Sensemaking in Realtime’ tool to gather observations, look for patterns and identify actionable insights.
The workshop is free. It is on 8 May 2020, from 12.00pm to 1.30pm AEST.
