How to evaluate a strategy: NSW Housing 2041 M&E Framework

Some public policy goals are so big and ambitious it is difficult even to comprehend their scope – let alone develop an approach to monitoring and evaluation that is credible, timely and useful. We designed an approach to do this and in the process, created a concept we come back to again repeatedly when designing monitoring and evaluation frameworks for strategies. 

The NSW Housing Strategy 2041 (Housing 2041, or the Strategy) is NSW’s 20-year vision for housing in NSW. It outlines the government’s long-term goals to deliver better housing outcomes for all people across the housing spectrum. Actions to achieve the 20-year vision will be updated over time. This approach provides flexibility and allows the NSW Government and its partners to update priorities based on changing circumstances, emerging evidence and community expectations. 

Delivering on the vision for Housing 2041 involves a significant investment of public funds. It is important that all funded programs, and the Strategy as a whole, are managed and delivered as efficiently and effectively as possible. The large scope and complexity of this whole-of-government approach calls for careful attention to evaluation.  

In 2023, DPE commissioned ARTD Consultants to develop a Monitoring and Evaluation (M&E) Framework for Housing 2041. 

The Department of Planning and Environment (DPE) needed clear guidance on how future housing actions align with, and are contributing to, the long-term goals of Housing 2041. In other words, the M&E Framework needed to provide a way of ensuring alignment between the actions being delivered and the Strategy’s intended outcomes and long-term vision. 

ARTD adopted a collaborative approach in developing the M&E framework, drawing on a range of key stakeholders across government agencies and the broader housing sector, including leaders and the Expert Housing Panel.  We wanted to make sure it was useful and would be used, and a big part of that was making it simple to implement despite a very complex policy area with fragmented data sources.  

In Q1, the ‘right direction’ refers to the intended outcomes of the strategy. Answering this question is relatively straightforward. It requires: 

• Clearly defined outcomes written in the form of a condition statement (i.e., who/ what is in what condition).
• Identifying metrics for each outcome. 
• Monitoring these metrics. 

In Q2, ‘things’ refers to the actions from the Action Plan. In other words, it asks if we are delivering our actions as intended. Answering this question is complicated because it requires monitoring diverse and difficult-to-measure actions. We can address this challenge using action rubrics. This approach provides standardised metrics that can be used for monitoring many diverse actions. These metrics can also be used to inform our selection of actions. This approach requires: 

• Understanding the logic of each action. 
• Developing an action rubric with a consistent set of criteria.
• Determining the levels of quality for Efficiency and Effectiveness, for each action.
• Rating each action against its rubric criteria.
• Collating these standardised metrics.
• Monitoring these metrics. 

Answering Q3 – Are we doing the right things? – is far more complex. The ‘right things’ are those that are feasible and most likely to be effective at addressing root causes of problems. These things are not always readily apparent. Outcomes are affected not only by what we are doing and how well we are doing them, but also by how other things we are not currently doing anything about are changing. Answering this question requires: 

• Reviewing all available metrics from Q1 and Q2. 
• Using subject matter expertise and creative thinking to inform strategic decision making.
  

Housing 2041 M&E Framework 

We developed an M&E Framework based on the core organising principle of a dashboard of data (Appendix 1) with a decision tree (Appendix 2). This provided a common set of metrics and a guide for decision-making based on these metrics for those leading the strategy. The idea is similar to a pilot with a dashboard of metrics, the data informs but does not replace judgement!

The decision-making process included thresholds for key conditions, such as specific values that, if reached, trigger a certain response such as in-depth research, evaluation or strategic decision making. These thresholds are like targets. But they are more easily updated as strategies change, and most importantly, imply corrective activities rather than simply identifying success or failure. Thresholds will trigger a response when outcome metrics, rubric ratings or risk ratings fall out of an acceptable range. Thresholds are effective governance mechanisms to support agile policy making. 

The Framework also supports the selection, design, development and implementation of actions to make sure they are aligned to strategic objectives. The Framework asks those responsible for actions to supply key information related to strategic objectives. This is done using evaluation rubrics – an approach that provides standardised metrics that can then be used for selecting and monitoring the actions. This creates an evidence-based approach to distributing funding to the actions. 

This project resulted in a useful and robust M&E Framework that DPE can use to guide the selection of initiatives, as well as manage efficiency and effectiveness risks to achieving the Strategy’s intended outcomes. The whole-of-government evaluation provided a systematic approach to assessing the impact of housing actions and identifying timely opportunities for government intervention. 

ARTD also developed implementation guidelines to support internal Housing Strategy staff and Agency Program Teams to establish and use the Framework. The involvement of the Expert Housing Panel underscored the government’s commitment to integrating both policy expertise and industry leadership into the housing strategy’s development, which stakeholders highly valued.

Importantly, we were able to develop an M&E Framework that can accommodate possible changes in direction necessary to achieve government objectives, despite the inherent uncertainty associated with a long-term strategy and the complex and dynamic context in which the Strategy and Framework operate. While the dedicated unit responsible for its development did not remain, it set a new standard for simplifying complex policy areas into clear, measurable outputs for ongoing improvement.
As the client noted:

‘…it allowed us to understand more effectively, what is a very complex system…… allowing government agencies to better respond to pressures with high impact policies and programs…. I only wish we could have kept it around long enough for it to see the light of day!” 

Below is a fictional prototype of a data dashboard and decision tree. It was created as a communication tool to support those leading the strategy.