Evaluation of Queensland Kids Partnership 2022-2025

Paul Ramsay Foundation

Project timeframe: 2022-2025

The Challenge

Queensland Kids Partnership (QKP) is an intermediary and coalition of leaders, organisations, practitioners and academics from across Queensland who work with the shared purpose of improving the conditions for children and young people to thrive. QKP exists to drive systems change, and to address key systemic barriers – political, structural, cultural and operational – to improving child wellbeing.

QKP was established in 2020, auspiced by the Australian Research Alliance for Children and Youth(ARACY). It is led and facilitated by a Backbone Team and governed by a Leadership Table (transitioned to an Intergenerational Stewardship Table in Phase 3), with members from some of QKP’s funding bodies, academic experts and non-profit and service provision organisations from diverse sectors aligned with child and youth wellbeing. This work is funded by a group of philanthropies and also receives funding from Queensland collaborators.

The Paul Ramsay Foundation (PRF), one of QKP’s funders, contracted ARTD and The University of Queensland’s Institute of Social Science Research (ISSR) to evaluate QKP Phase 2 between July 2022 and June 2025.

QKP were also conducting an internal evaluation in parallel.

The Objective

The project was commissioned as an evaluation balancing two purposes: learning for QKP and accountability to PRF. This is not an uncommon challenge in philanthropically funded work. The intention of the evaluation was to learn about what works in engaging multi-sector organisations around common goals and values, building the knowledge, networks and capacity of the sector, and in collective advocacy efforts for policy and systems change.

The evaluation took a developmental approach and was highly systems-aware, drawing on systems change, intermediary/field-building and systems thinking literature to inform the design of the evaluation and the contextualisation of the findings.

We have conducted several evaluations of systems change interventions and they all have their unique challenges – often centred on balancing learning and accountability. This is in a context in which clear evidence of long-term outcomes attributable to interventions is challenging. This is why we always take a theory informed approach and pay attention to the levels and boundaries of any given system, and hold our evaluation team and collaborators to a high level of critical thinking (especially where it comes to questions of causality).

Our Approach

While we initially began with an evaluation plan for the whole period, in reality, the evaluation was delivered in 2 discrete stages. This approach emerged in response to the developmental stage QKP was at when the evaluation was planned, and the changing needs of QKP and PRF during the first year of the evaluation.

As we noted in our learning webinar with the Paul Ramsay Foundation and QKP, one of the lessons from this work was around the most useful role for evaluators in a formative phase of a systems change initiative like QKP. When we were commissioned to undertake the evaluation, we were tasked with developing a plan to evaluate QKP. However, at the time there was a solid core, all the evidence and plans – but relationships were still forming, team members were still being onboarded and because everything for QKP is done collaboratively, some of the details of initiatives were in early stages of being co-designed, or weren’t yet at that stage. Processes and technology were still being set up, and there weren’t yet solid plans around what sorts of routine data collection there would be to inform our evaluation planning. QKP’s internal monitoring and evaluation framework for their internal evaluation (which ran parallel to this one) was also still being developed, and timeframes for their data collection activities had not yet been planned. This context informed the pivot to the more staged approach.

Stage 1

Stage 1 of the evaluation (January 2024 to June 2024) focussed on how QKP was engaging the system and improving evidence. We began by building the evaluation framework and plan, which aimed to capture how the 11 Initiatives were achieving QKP’s overarching objectives and where they were seeing influence on the system levers they had identified as key to systems change. As a systems change initiative, QKP were already using The Waters of Systems Change framing to think about their work. We integrated this framework, and Donella Meadow’s levers of systems change into how we thought about QKP’s work and how to evaluate the change they were making.

The mixed methods approach included:

interviews with decision makers from the many sectors QKP engages
interviews with QKP Backbone Team members, and QKP’s partnership survey
Bellwether method interviews, which were designed by the Harvard Family Research Project to find out where on the agenda a policy issue is situated, by interviewing those who have the potential to influence its position on the agenda. These interviews were conducted by our partners at ISSR, at a remove from the evaluation
an extensive review of documents and data (120+ sources), which used a document review tool and thematic coding framework tied back to the rubric dimensions for analysis
use of data from a partnership survey distributed by QKP for their internal evaluation.

We also undertook regular meetings to talk through what was happening at QKP, and identify emerging data sources. We also co-located at the QKP offices for several days to support internal monitoring and evaluation activities, and exchange ideas on the parallel evaluations.

We held a sensemaking meeting with QKP Backbone Team members and philanthropic partners to explore and contextualise findings, before delivering the report.

We aligned the findings and recommendations of the report with the Waters of Systems Change framework, linking the ‘what’s so’ and ‘so what’ of the evaluation with the systems theory.

Stage 2

Following a review with PRF and QKP of the previous stage of the evaluation, we collectively agreed there was a need for some evaluation capability building ahead of Phase 3. This involved supporting QKP’s backbone team, funders and partners to develop a more granular theory of change, KEQs and indicators to use in monitoring and evaluation of Phase 3, which had received the green light from QKP’s group of funders. During this period, we also refined the rubric developed in Stage 1 of the evaluation, for use in Stage 2 as well as for QKP to adopt in their own evaluation activities in Phase 3 of the evaluation.

We then delivered a second evaluation for the period December 2024 to October 2025 focussed on identifying emerging evidence of sustainable change in the system. This mixed methods evaluation involved:

a second round of interviews with decision makers (n=8), interviews with partners and collaborators (n=19)
focus groups with (n=12) Backbone Team members
analysis of QKP’s partnership survey (n=53)
review of supporting data and documents.

To support synthesis and judgement, alongside thematic analysis of qualitative data, we utilised the rubric developed with QKP’s Backbone Team to support an understanding of QKP’s systems goals, and their degree of maturity.

We held another sensemaking workshop was held with QKP Backbone Team and philanthropic partners to validate the findings and rubric scores.

The Impact

Impact of Stage 1

The evaluation’s findings and recommendations were used by QKP in their strategic planning for Phase 3 of the initiative, and to inform the successful funding proposal for a further 5 years. A key recommendation to increase strategic communications capacity was taken up with the employment of a new staff member.

The findings as well as what ARTD, PRF and QKP learned about being involved in an evaluation of this kind of project were shared in a webinar for the sector, which can be viewed here: Webinar: Enhancing multi-sector engagement – Queensland Kids Partnership

The evaluation report from stage 1 is available here

Impact of Stage 2

The work we did with QKP’s backbone team, funders and partners helped to:

refine some elements of the Phase 3 strategy
strengthen QKP’s monitoring, evaluation, learning and reporting framework
produce a shared understanding between QKP’s backbone team, partners and importantly funders of what system conditions and population level changes QKP could reasonably be expected to contribute to, and how these could be measured.

Our work together with QKP and ISSR, and the collaborative learning approach we built together also resulted in greater evaluation capability for all those involved.

The final evaluation report found that QKP had successfully set the conditions to progress in the next 5 years to begin impacting outcomes for children, youth and families. Findings and recommendations were used to strengthen ongoing planning for implementing Phase 3.

The evaluation also resulted in further opportunities to do deep thinking about systems change initiatives and their evaluation, and how to approach it, resulting in this blog: How I ate an elephant – evaluating system change in bite size chunks | ARTD

The evaluation deliverables have also added to the available information out there on how to evaluate systems intermediaries!

The Stage 2 evaluation report can be found on the PRF website here.

Additional details

Evaluating a systems change initiative is not straightforward. Evaluating an intermediary organisation that works as a collaborative with partners is even less so, as Mark Cabaj and many others have noted in their work[1].

There were many lessons we took as an evaluation team from this project and from working with our colleagues at QKP and ISSR. We are grateful to all involved for their dedication to building strong working relationships that allowed us to challenge each other’s thinking, and produce a stronger evaluation and implementation learnings. It’s commendable that the lessons learned in conducting the evaluation have also been shared.