Connecting Evaluation and Indigenous Knowledges for Country with Practical Action – session with Associate Professor Megan Williams
Kate and Ken recently attended an interactive Australian Evaluation Society session with accomplished academic, poet and Wirajuri woman, Associate Professor Megan Williams on Connecting Evaluation and Indigenous Knowledges for Country with Practical Action.
Associate Professor Megan Williams is the author of the Indigenous evaluation framework Ngaa-bin-ya, which was published in the Evaluation Journal of Australasia (EJA) in 2018 and was the most downloaded EJA article of that year. It builds on Stufflebeam’s (2003) ‘Context, Input, Processes, Products (CIPP)’ model. It can be used by evaluators to better determine how Aboriginal and Torres Strait Islander people are impacted by a particular program or policy and what future implications it might have (as Megan noted, around a third of Aboriginal and Torres Strait Islander people are under 21, a key piece of context for future implications of evaluations).
The session walked us through the seven key steps of the Ngaa-bin-ya process which include:
- Forming an evaluation Reference Group, working group or team
- Going through prompts and identifying those that are relevant to this project and the local context
- Identifying data to answer research questions and prompts
- Obtaining ethics clearance
- Collecting and cleaning data
- Going back through the prompts as a group and discussing them for analysis, interpretation and reporting
- Writing up findings according to prompts or themes arising from data or program logic.
Megan also stepped us through the four domains of Ngaa-bin-ya – landscape, resources, ways of working and learnings – and a checklist of evaluation considerations in each domain to work through with an Aboriginal Reference Group.
Hearing Megan speak made us consider how aligned with the Ngaa-bin-ya framework our evaluation of the Future Directions Service Improvement Initiatives for the NSW Department of Communities and Justice (DCJ) has been.
This evaluation is being led by an Aboriginal Reference Group, which includes a diverse group of Aboriginal elders and Aboriginal program stakeholders. We’ve worked with our Reference Group to identify the locations where we should focus our fieldwork, the key stakeholders to engage at each location and the best way of engaging with them. Our Aboriginal Reference Group has led us in thoughtful discussion about the emerging evaluation findings and has helped us anchor our recommendations within the broader Aboriginal Affairs policy context.
The Future Directions evaluation has focused on the extent to which Aboriginal people are aware of and can access a range of housing products and services. Knowing that we required ethics clearance from the Aboriginal Health & Medical Research Council of NSW (AH&MRC) we set up an Aboriginal Reference Group to oversee our evaluation. To date, we have had seven reference group meetings.
Megan also wove poetry (her own and others’) seamlessly into the session, starting with ‘Which way Asimov?’ from the Gomeroi poet and legal researcher Alison Whittaker’s book Blakwork. By using poetry, a form of self-reflection and personal expression, alongside her reflections on the ways in which evaluation and its processes have intentionally or otherwise excluded Aboriginal and Torres Strait Islander people, lives and worldviews, Megan created a session that engaged non-Indigenous attendees in understanding the ways in which evaluation is felt at the individual and family level. It was a reminder that evaluation is political but also personal. It added an emotional dimension of knowing to the intellectual understanding we have that non-Indigenous evaluators must do the work to understand and incorporate Indigenous frameworks, history, and people in work that affects Indigenous lives. She reiterated the strong connection with country than many Aboriginal have and the importance of including this in any evaluation involving Aboriginal people.
We will certainly consider utilising the Ngaa-bin-ya framework again in future.
