Indicator and Outcome Bank

The increase in natural disaster frequency and severity across NSW presents new challenges for how NSW collectively prepares for, responds to and recovers from such events, and how it measures the effectiveness of its actions. Many kinds of programs and activities contribute to disaster recovery, resilience, preparedness and risk reduction. Therefore, policy- and decision-makers need to be able to understand and compare what works and for whom, how and when. This requires clear articulation of intended outcomes and consistent, available data against which those outcomes can be measured, where possible. 

The Overarching Bushfires Evaluation reviewed 12 major NSW Government recovery programs in response to the 2019-2020 bushfires. One of the key learnings from this evaluation was that program data collection was inadequate to assess value for money and that evaluation should be embedded in program design. Even where there was reasonable evidence (e.g. from the literature or evaluation reports) to suggest certain outcomes had been achieved, these could not be quantified without evidence of the number of program participants experiencing change and the quantum of benefit received. A further challenge was the lack of baseline data, data to establish a counterfactual (what would have happened in the absence of programs) and the ability to attribute outcomes where program overlaps existed.  

In particular, the evaluation made 2 recommendations that led to this project:  

• Design data collection frameworks for outputs and outcomes on a community basis, considering feasibility and ethics. Recognise that some data cannot be ethically or feasibly collected in disaster recovery.  
• Communicate data needs upfront. Set reporting requirements clearly so data collection can be considered in project design. 

We were commissioned to develop a comprehensive yet strategic repository of outcomes, indicators and metrics (the Bank).  

The Bank is intended to be used by the NSW Reconstruction Authority and by agencies who receive funding to implement disaster recovery, preparedness, risk reduction and resilience programs. The Bank will help them to:  

• design programs, by providing a clearer understanding from the outset of what programs should achieve  

• design and develop evaluation frameworks and implement evaluations by articulating how outcomes can be measured and from where evidence can be sourced. 

The Bank seeks to improve the consistency of outcome definition and performance measurement by DRFA administering agencies and the NSW Reconstruction Authority. 

We partnered with Nicki Hutley Economics and established a Project Advisory Group with staff from the NSW RA and DRFA-administering agencies in NSW, such as Department of Planning, Housing and Infrastructure, Transport for NSW and Legal Aid NSW.  

We conducted a thorough desktop review of >1000 existing metrics and outcomes currently used by the NSW Reconstruction Authority, the National Emergency Management Agency (NEMA), the Australian Institute for Disaster Resilience and identified NSW agencies, including the NSW Treasury’s Performance and Wellbeing Framework and Outcomes Values Database. These metrics were assessed for duplication, relevance and feasibility, and distilled. 

Interviews were conducted to understand stakeholder’s monitoring and evaluation design and planning shortcomings, and how the Bank could support them. The Project Advisory Group peer reviewed the first consolidated list of outcomes and indicators, and Project Advisory Group meetings were used to repeatedly sense-check progress on the Bank and its related documentation. 

Believed to be the first of its kind in NSW the Bank contains 17 outcomes across the 4 recovery domains (Built, Economic, Environmental, Social), 80 indicators, 110 metrics, proposed data sources for metrics, and notes that included if metrics are suitable for inclusion in Cost Benefit Analyses. The Bank also mapped how each of its outcomes aligned to the NSW Reconstruction Authority’s strategic goals. 

We also developed a Bank User Guide, which helps to navigate, interpret and use elements of the Bank, and a Change Implementation Plan that the NSW RA will use to promote, track and improve Bank use and useability. Feedback from Project Advisory Group members was resoundingly positive. We were reengaged by the NSW Reconstruction Authority to use the Bank to develop a monitoring and evaluation framework for a suite of recent programs. The Bank has also since been used to inform a concurrent evaluation ARTD is working on of NSW Reconstruction Authority grant programs.