Families Capacity Building Project Evaluation

ARTD was commissioned to design an Evaluation Framework for a national program, Families Capacity Building (FCB), funded via the National Strategy to Prevent and Respond to Child Sexual Abuse.[1] The program is both universal (‘Online safety is for everyone’) and targeted to parents and carers of children who can experience greater vulnerabilities to online risks, including children who are culturally and linguistically diverse, Aboriginal and Torres Strait Islander, with disabilities or developmental delays, are in out-of-home care, or are in LGBTQIA+ families. A key activity of the program is to build relationships with intermediary organisations to reach the target groups.

The overall FCB program comprises diverse components targeting different levels, groups, and outcomes, and therefore the Evaluation Framework needed to focus on the granularity of the program components, as well as how they were working together to achieve outcomes at a programmatic level. Although the program was funded until 2025, the activities were expected to continue beyond this date, so we designed the Framework to be useful over the longer term. It was important for the Framework to include guidance on conducting the FCB evaluation in an ethical and culturally safe manner with the target groups. There is a lack of evaluation evidence for online safety programs, so the Framework needed to include robust methods to help build the evidence base. A further requirement for the Framework was to ensure alignment with the funding agency (National Office of Child Safety) for monitoring and evaluation of measures funded under the National Strategy.

The program had many components and was still being designed when we were engaged, so a developmental approach was required. Based on our review of the existing data available to the FCB program team and their preferences for data collection methods, we recommended a mixed methods design.

We worked with eSafety to refine the program theory. To facilitate this process, we updated a literature review of relevant theories and evidence from psychology, communications and public health. We conducted a program logic workshop involving the FCB program team, eSafety evaluation team and other stakeholders, which generated thoughtful discussion and resulted in a well-developed logic model and key evaluation questions.

After documenting the available data sources and recommending additional data collection, we linked data sources with indicators that could be used to determine whether program objectives had been achieved. Indicators were categorised into two types, corresponding with the impacts defined in the FCB theory of change: reach and awareness (the extent to which the program had connected with priority populations and the broader population) and behaviour change (the extent to which the program activities had desired effects on parents’ and carers’ willingness to talk about and take action against online child sexual abuse). Not all the key evaluation questions could be linked with quantitative indicators, and for these we recommended a qualitative approach.

Internal stakeholders and external organisations who work with and represent the target groups were involved in a journey mapping workshop. The purpose of this was to provide insights into how the program could effectively reach its target audience and how they might be expected to respond, as well as to identify points at which critical processes and outcomes could be measured. The workshop also provided an opportunity for the FCB team to engage with key stakeholders and involve them in the design of the program and its evaluation.

In a separate workshop with eSafety staff, we developed a risk register to identify potential risks relating to the FCB program and its evaluation. This was designed to be a ‘living document’ that could play a functional part in the delivery of the program, to be discussed and reviewed regularly, allowing for dynamic and ongoing management of risks.

All these components were included in the Evaluation Framework, in addition to background information, a program overview, a description of evaluation data sources and measures, guidance on governance, data analysis, and reporting. and data collection materials.

We later provided advice and guidance on a draft ethics application developed by the FCB team and evaluation staff of eSafety to conduct the evaluation using the Framework.

The FCB team received ethics approval for the evaluation and, as suggested in the Evaluation Framework, they have developed an evaluation plan. They are now working towards its implementation as many of the FCB program resources are available on the eSafety website (see: Keeping children safe online in communities | eSafety Commissioner)