2021: a reflection on lessons learnt
In some ways, 2021 has felt never ending. In others, it has gone past in the blink of an eye. For our final blog of 2021, we asked our ARTD team to reflect on what has shaped our evaluation thinking and practice most this year.
Jade
This year has reminded me how important it is to have a plan but be ready to throw it away when circumstances change, to not take it as a given that people will see the value that evaluation can provide, and to have the input of a team who can see the things you cannot, in particular: people with lived experience. Evaluation is at its best when it’s connected and responsive to the (often changing) reality of our clients and their clients.
Emily
With several large projects coming to an end in 2021, I’ve been reflecting on what I know now that I didn’t know when the projects started; and what I’d do differently next time. I’ve realised that large, multi-year projects cannot—and arguably, should not—look the same at the outset as they do at the conclusion. Holding fast to an agreed plan can be important, but so is flexibility when the context shifts (and hasn’t it shifted this year!). Creativity is also crucial: asking ourselves where we can find good quality information in places we hadn’t planned to and analysing it thoughtfully to add value.
Ken
2021 for me has been about constantly having to adapt to COVID-19 and the surprises and challenges it throws up. Whether that be working remotely from home, changing project plans and timelines with clients or working out how to best engage stakeholders, a healthy dose of flexibility and resilience have been required. It’s also been interesting to use Qualitative Comparative Analysis (QCA) as a method for the first time in our evaluation of the Entrepreneurship Facilitator initiative.
Moya
A short speech from Margaret Crawford, the current Auditor-General of NSW, at this year’s FestEVAL stands out to me as a memorable moment that shaped my thinking on evaluation this year. For context, the Auditor-General for NSW helps the Parliament hold Government accountable for its use of public resources. This includes performance auditing of NSW government departments or agencies, many of whom are our clients. It made me wonder, what is the relationship between performance auditing, evaluation and accountability? And, what does it mean for us as evaluators? Performance audits cannot comment on the merit of government policy—only how well it is implemented—but evaluators can. The Auditor-General cannot report back to the government agencies, but we can. What I’ve realised is that, as evaluators, we sit in a privileged and unique position which allows us to help policy makers and program staff to look forwards for learning rather than backwards for accountability. To me, this is how evaluation adds value and best helps our clients.
Kate
I’ve been really enjoying reflecting on the ideas in Luke Craven’s Pig On The Tracks newsletter and how much these apply to evaluation. A couple that stuck out were the idea that no tool (method, approach) is omni-competent, and that you need a toolbox of tools (and a rack of soft skills) to deal with complexity (#36). Also, that no principle can be applied to everything either (see the pattern?) (#38). My final takeaway from the newsletters is that pooh-poohing reductivism can be just as much a mistake as thinking one can make a simple model or tool that will work in the real world in many different contexts. Reflecting on David Graeber’s work, Craven wrote “We so commonly write simplification off as a form of stupidity, but it can be a form of intelligence. It’s an initial move, from which more sophisticated ones can follow.” (#34).
Keely
Joining ARTD this year, it was interesting seeing how evaluation has changed and evolved with COVID. This year slapped us all in the face with the reminder to be kind to others but also to ourselves. It has provided me with an opportunity to develop greater empathy, the patience to listen and to be flexible. While it has been a strange year, ultimately it has allowed me to grow and hopefully make a little bit of difference in this complicated world.
