The Australian Evaluation Society’s FestEVAL: Four unique and compelling provocations
Evaluation in a complex world requires us to take a ‘Blue Marble’ perspective, understanding what’s occurring globally to understand what’s happening locally. Michael Quinn Patton
Last week, evaluators from Australia and the Pacific gathered for the Australian Evaluation Society’s (AES) FestEVAL: a week-long program (online, of course) of activities and webinars to celebrate evaluation, create connection between evaluators and to challenge our thinking.
It was a rich program. On reflection, the presentations that stuck with me most were the four very different—yet equally compelling—provocations given in the FestEVAL opening address.
Provocation One from Simon Kuestenmacher: Pay attention to demographic trends
Simon, Co-Founder and Director of The Demographics Group, argued that the Australian ‘middle class’ is a fanciful, romantic notion, left over from the 1970s. Using 2016 Census data, Simon unpacked interesting and revealing insights about the new ‘working class’.
There’s a correlation between education, income and spending. Australian occupations are classified into five skill levels. Simon’s analysis shows the share of growth in new jobs is greatest for Skill Level One (jobs requiring the most formal training, such as engineers or doctors) and Level Five (jobs requiring no training, such as wait staff). So, growth in the Australian labour force is ‘U’ shaped; where the rich are becoming richer and the poor are becoming poorer.
He argues that a socially cohesive country is near impossible with this disparity. This picture of the Australian labour force will affect all projects and services we evaluate and must be considered in our work.
Provocation Two from Nicole Tujague: Indigenous program evaluations need to be Indigenous led
In 2016, a report by Sara Hudson found that Australia spends $5.9 billion on programs for Indigenous people, but less than 10% of those were being evaluated. Nicole is a descendant of the Kabi Kabi nation from Mt Bauple, Queensland and the South Sea Islander people from Vanuatu and the Loyalty Islands. She argues that we do not evaluate Indigenous programs enough and that when we do, we do them wrong by not commissioning Indigenous evaluators.
Nicole says that, as evaluators, we need to give up the power and let Indigenous people lead the evaluations of the programs that affect them.
Provocation Three from Eleanor Williams: We need to move faster to be relevant
If there’s one thing COVID–19 has taught us, it’s that we need credible information quickly to inform decision-making. Eleanor, who leads the Centre for Evaluation and Research Evidence at the Department of Health and Human Services, argues that we evaluate too slowly to influence design and change. She reminds us of the ‘iron triangle’, where speed, scale and cost are traded off. She provokes that evaluators often consider method and scale, but do not consider speed.
Eleanor asks us to consider three questions:
- What is holding us back and what could we lose by evaluating faster?
- In what situations would faster evaluation work best?
- What types of programs lend themselves to rapid evaluation approaches?
Eleanor acknowledges that going faster 
could affect findings and the depth of insights in certain situations, and that program maturity (and hence, time) is required for many outcomes evaluations. She encourages us to go faster when it is right to do so, particularly when there is sufficient evaluation funding, stakeholder support for rapid evidence generation and when the program is being implemented quickly and is reasonably simple.
If we evaluate faster where possible, our relevance in the policy process will increase as we are able to keep pace with policy decision-making. In a follow-up debate session about this, a poll among participants suggests that most of us agree we can go faster, though for context most people think we needed a balance of fast and longer-term evaluations.
Provocation Four from Michael Quinn Patton: There is no return to normal
Michael, a leading thinker and very well-known author and evaluator, started his provocation by painting a picture of the very complex current world we live in. With the Covid-19 pandemic, coupled with social justice uprisings, the climate emergency, and the age of ‘infodemia’, Michael argues we need to bring a ‘Blue Marble’ perspective to our work. In other words, we need to understand what is occurring globally to properly understand what is happening locally.
As evaluators, and part of the larger scientific community, we contend with audiences who increasingly want to hear what they want to hear. Quinn Patton argues that evaluators need to think in systems and work across the boundaries of sectors and projects and, in doing so, move away from generalisable evaluation models. If we want to move forward as a profession, we need to adapt and pivot in this changing world of complex systems.
As I reflected on these provocations, I realised that they come with both advantages and disadvantages for a young evaluator. Sure, I lack the experience and depth of knowledge so critical to our work, but I come to evaluation with no preconceptions. What constitutes new and challenging realities for experienced evaluators are the only realities young evaluators know. It reminded me of the benefit of working in multi-generational and diversely skilled teams, and the importance of always questioning our approaches to evaluation in the complex and changing world we live in.
I fell into evaluation by accident, coming from a background in Law and Communications. Last week I learnt that I am not alone, and that it is the very diversity of backgrounds and skill sets that makes the evaluation field so enriching.
