Program logic, monitoring and evaluation plan for the NSW Health Safety and Quality Essentials Pathway

The Clinical Excellence Commission (CEC) is one of the pillars of NSW Health, charged with leading, supporting and promoting improved safety and quality in clinical care across the NSW health system. As the lead agency in this space, CEC has been delivering education and training as part of the ‘toolkit’ to improve safety and quality since 2007.

The CEC’s education program is undergoing significant reform, with the 2021 launch of the state-wide Safety and Quality Essentials Pathway. The Pathway will provide educational resources relevant to every NSW Health staff member from their first day onwards, stimulating discussion and shared accountability through a common language and mindsets about quality and safety in practice.

ARTD worked with the CEC and external stakeholders to develop a program logic and monitoring and evaluation framework for the NSW Health Safety and Quality Essentials Pathway. The framework provides a blueprint for ongoing monitoring and a phased evaluation, over an indicative six-year period.

As part of the Framework design process, ARTD undertook consultations with CEC and external stakeholders including workshops to refine the evaluation questions, program logic and outcomes matrix, as well as interviews and meetings at key points.

The Safety and Quality Essentials Pathway is a significant reform being implemented into a large, complex, human system. As such, we drew on systems thinking to inform the selection of the methodology for this evaluation. A case study methodology was selected as the most appropriate for the context, with the ‘case’ being the NSW Health entity service system.

The mixed-methods evaluation design also draws from several behavioural and organisational change theories (normalisation theory and systems thinking) and from different schools of thinking about evaluation:

• Normalisation theory informs both the theory of change – how it is expected that the Pathway may work – and measurement of the extent to which this has eventuated.
• A developmental evaluation approach has been incorporated, which acknowledges that the evaluation is designed to support a program that is being iteratively developed, with a focus on real-time data collection and systematic reflection to support ongoing learning.
• A realist evaluation lens ensures context is considered at all stages of the evaluation.
• A cost consequence analysis will be used to provide an economic appraisal that complements the outcomes evaluation to give another perspective on the value or worth of the Pathway.

The Framework matches the phases of the evaluation to the roll-out phases of the Pathway, incorporating an implementation review, process evaluation, outcomes and economic (cost-consequence) evaluation.

The CEC will establish governance procedures and internally socialise the Framework with stakeholders across NSW Health.

The CEC will lead and partner with the system in monitoring the performance of the Safety and Quality Essentials Pathway over a six-monthly cycle for the period of implementation and one to two years following. Measures have been selected to provide regular signposts about the extent to which the implementation of the Pathway aligns with the design intentions and outcomes of the work. For example, reach of the training and policy-level outcomes related to patient safety.