Promoting the use of Digital Technologies in Evaluations
In late 2020, as part of their Evaluation Learning Sprint series, I attended an online Australian Evaluation Society (AES) professional learning seminar on ‘Digital Evaluation’ presented by Dr Samantha Abbato, Director of Visual Insights People and Margi MacGregor of CatholicCare NT.
The value of digital has certainly been hammered home during the global pandemic – without it, encouraging stakeholder engagements would have been much more difficult! But Samantha and Margi raised many compelling reasons to incorporate elements of digital into evaluation engagement strategies and deliverables:
- Visual content such as videos, photos, charts and infographics are often more memorable than just words on a page when used well. They can also be easier to remember, relate to or understand: research shows that individuals process visuals 60,000 times faster than words or verbal communication.
- Using visuals can be a great way of supporting a participant-centred approach, featuring and platforming the voices of participants in contrast to other data collection methods such as surveys.
- Capturing the ‘story’ behind a particular program, policy or organisation’s existence visually, can be useful for developing a theory of change or program logic diagram: Samantha referred to Funnell and Rogers’ work on Evaluation Theory and how using story or the inductive way of developing a Theory of Change is a strong method for recognising people.
- So much of the work of strategy setting and evaluation is about communication. Using digital technologies to engage people who learn and retain information better visually can only help to strengthen communications and make them more inclusive.
- Digital data capture can offer more inclusive opportunities for participation (for example, someone who cannot read and write English well might be willing to have their feedback recorded or filmed and still can participate in an evaluation).
- Most evaluations benefit from a mixed methods approach involving qualitative and quantitative methods, offering many opportunities to integrate visuals into data collection, analysis and reporting processes.
- Tools for digital engagements (for example, webinar software, online whiteboards and video engagement software), and the ability to produce, edit and publish visuals and video content are rapidly increasing in accessibility, ease of use and quality. Sometimes they can even reduce the costs of engaging participants as travel might not be needed!
Challenges
Incorporating digital and visual approaches in our evaluations might not always be easy and as with any new approach, you may encounter speedbumps in implementation! Some challenges discussed included:
- resourcing requirements: adopting digital approaches may require significant time and investment in new devices such as iPads and digital hubs.
- connectivity issues: creating new digital content such as large video and audio files requires high internet speeds and bandwidth which may be a challenge in regional and remote areas (Samantha and Margi mentioned how they helped create a policy where large data files were uploaded to a shared platform outside of business hours to reduce pressure on the existing connection).
- evaluator reluctance: members of an evaluation team may be inexperienced using digital technologies as part of an evaluation and may be reluctant to start learning new tools or software until they have been properly trained.
- participation reluctance: some stakeholders may not feel comfortable speaking in front of a camera.
If in doubt, it can be best to start small. As Margi suggested, we should consider switching ‘bit by bit’ by incorporating elements of digital before jumping in to undertake an entire evaluation using digital approaches. Some suggestions for where we might like to begin are:
- using more charts and images in our deliverables to replace text
- developing digital dashboards to collect and present data and projections
- including links to online resources, videos, and interactive platforms in our reports
- filming or recording client/ stakeholder/ customer/ beneficiary case studies.
Reflecting on the process
Samantha and Margi had some words of advice for others considering integrating more visual approaches into their evaluations.
While developing a shared vision throughout the evaluation team or organisation is key for any evaluation, it is particularly essential for digital evaluations, to ensure everyone is on the same track and challenges and concerns can be raised early and worked through: before there’s been a large investment of time or dollars in a digital output. Regular and ongoing diagnosis of progress and staff readiness is also encouraged.
To ‘prepare for the journey’ they recommended ‘seeing it as a major organisational change that needs to be managed appropriately.’ This may require education and training to ensure evaluation team members, participants, support staff and other stakeholders are well supported to integrate digital data collection and evaluation approaches into an organisation (and confident in doing so).
Finally, it’s important to choose the right tool for the job! Approach digital engagement and evaluation as you would any other method, and choose appropriately for your stakeholders, the evaluation, the stakeholders involved and the available resources. It doesn’t matter how glossy the production value is if your participants don’t feel included comfortable with the engagement methods used!
The recording of the online seminar is available here.
