Health
Program logic, monitoring and evaluation plan for the NSW Health Safety and Quality Essentials Pathway
Clinical Excellence Commission, 2021
THE CHALLENGE
The Clinical Excellence Commission (CEC) is one of the pillars of NSW Health, charged with leading, supporting and promoting improved safety and quality in clinical care across the NSW health system. As the lead agency in this space, CEC has been delivering education and training as part of the ‘toolkit’ to improve safety and quality since 2007.
The CEC’s education program is undergoing significant reform, with the 2021 launch of the state-wide Safety and Quality Essentials Pathway. The Pathway will provide educational resources relevant to every NSW Health staff member from their first day onwards, stimulating discussion and shared accountability through a common language and mindsets about quality and safety in practice.
THE OBJECTIVE
ARTD worked with the CEC and external stakeholders to develop a program logic and monitoring and evaluation framework for the NSW Health Safety and Quality Essentials Pathway. The framework provides a blueprint for ongoing monitoring and a phased evaluation, over an indicative six-year period.
OUR APPROACH
As part of the Framework design process, ARTD undertook consultations with CEC and external stakeholders including workshops to refine the evaluation questions, program logic and outcomes matrix, as well as interviews and meetings at key points.
The Safety and Quality Essentials Pathway is a significant reform being implemented into a large, complex, human system. As such, we drew on systems thinking to inform the selection of the methodology for this evaluation. A case study methodology was selected as the most appropriate for the context, with the ‘case’ being the NSW Health entity service system.
The mixed-methods evaluation design also draws from several behavioural and organisational change theories (normalisation theory and systems thinking) and from different schools of thinking about evaluation:
- Normalisation theory informs both the theory of change – how it is expected that the Pathway may work – and measurement of the extent to which this has eventuated.
- A developmental evaluation approach has been incorporated, which acknowledges that the evaluation is designed to support a program that is being iteratively developed, with a focus on real-time data collection and systematic reflection to support ongoing learning.
- A realist evaluation lens ensures context is considered at all stages of the evaluation.
- A cost consequence analysis will be used to provide an economic appraisal that complements the outcomes evaluation to give another perspective on the value or worth of the Pathway.
The Framework matches the phases of the evaluation to the roll-out phases of the Pathway, incorporating an implementation review, process evaluation, outcomes and economic (cost-consequence) evaluation.
IMPACT
The CEC will establish governance procedures and internally socialise the Framework with stakeholders across NSW Health.
The CEC will lead and partner with the system in monitoring the performance of the Safety and Quality Essentials Pathway over a six-monthly cycle for the period of implementation and one to two years following. Measures have been selected to provide regular signposts about the extent to which the implementation of the Pathway aligns with the design intentions and outcomes of the work. For example, reach of the training and policy-level outcomes related to patient safety.
_______
Strong Minds Program Evaluation
Western NSW Primary Health Network, 2019
THE CHALLENGE
The 2014 National Mental Health Commission Review of Mental Health Programmes and Services highlighted the complexity, fragmentation and inefficiency of the mental health system, and in doing so presented a strong case for reform. The Australian government response recognised the need to implement an ambitious plan for change to reduce the health, social and economic costs of mental illness and suicide for individuals and the community. The new approach to mental health funding includes a focus on person-centred care, with stepped care identified as a priority reform to support redesign of primary mental health care programs and services. This entails moving from a ‘one size fits all’ approach to services matched to individual consumer needs.
The Strong Minds program, funded by Western New South Wales Primary Health Network and delivered by Marathon Health, replaced previous psychological and mental health service models in Western NSW. ARTD was commissioned to evaluate Strong Minds, which aims to provide psychological therapy services for under-serviced groups, including: people living in rural and remote communities; children younger than 12 years; people experiencing, or at risk of, homelessness; women experiencing perinatal depression; and people from culturally and linguistically diverse backgrounds. It also targets people from designated high-needs population groups (such as Aboriginal and Torres Strait Islander people, and people who are at risk of suicide).
THE OBJECTIVE
ARTD undertook a mixed-methods evaluation to determine whether Strong Minds had met its intended aims and objectives and to inform judgements of value for money. The evaluation drew on a range of existing and newly collected data, including semi-structured interviews with Marathon Health staff, referral partners and other stakeholders, an online survey of Strong Minds clients and general practitioners, and quantitative analysis of Strong Minds referrals data and service and outcomes data for current and former Strong Minds clients.
IMPACT
The evaluation showed the Strong Minds program reduced psychological distress of people with moderate to severe mental health issues, and that there were a range of positive outcomes for participants, including an improved understanding of their mental health, connection with relevant community supports, and re-engagement with family, education or employment.
Strong Minds was shown to complement other supports and services for people with mild to moderate mental health illness within a stepped care model.
The evaluation also highlighted many areas of effective delivery as well as opportunities to improve the program’s implementation and reach.
_______
Snapshot evaluation of NSW Health Management Development Programs
Health Education and Training Institute (HETI), 2020
THE CHALLENGE
NSW Health is an employer for more than 140,000 people in NSW. Developing and enhancing leadership and management skills continues to be a key strategic priority, ensuring that NSW Health employees who are responsible for the management of people and programs have the appropriate skills and knowledge to be successful in these roles.
The Health Education and Training Institute (HETI) has a critical role in achieving these strategic priorities, providing a range of management development offerings that increase the knowledge and skills of NSW Health employees throughout their career trajectories.
THE OBJECTIVE
ARTD worked with the Health Education and Training Institute to provide a high-level, snapshot evaluation of four major education products that deliver leadership and management development programs to all NSW Health staff.
The evaluation was designed to maximise the value of the extensive existing data on the education products, to better understand their strengths and weaknesses and inform future decisions about delivery.
OUR APPROACH
HETI had extensive participant feedback data, collected via survey across four years of delivery of the programs, containing both quantitative and qualitative responses.
ARTD undertook data analysis of the quantitative feedback data on each program, and also looked at insights emerging across the programs. To do this, we utilised R and the RStudio statistical software, which allowed us to develop code that rapidly produced crosstabulations of each of the various measures of outcomes against different cohort profiles (for example, organisation types or professional groupings).
We partnered with Altometer Business Intelligence to provide qualitative analysis of free text comments using machine learning. Insights were synthesised to help inform and explain the quantitative data findings – particularly where we found difference between groups.
Using a combination of machine learning, statistical analysis code developed in R and dedicated teams, we were able to deliver extensive and useful findings in a very compressed timeframe of less than six weeks from start to finish.
IMPACT
We were able to demonstrate the impact of the programs as reported by participants, backed up with both qualitative analysis and descriptive statistics supported by statistical significance testing. We were further able to show clearly which cohorts across a range of demographic profiles were engaged differently providing useful insights for targeting promotion and consideration when making changes.
The report has been used to inform funding discussions and quality improvement processes for the educational products.
_______
Review of District HETI Relationship Model
Health Education and Training Institute, 2017.
THE CHALLENGE
District HETI was established in 2012 with a remit for the rapid development of online education and training for the NSW Health workforce. This was a time of substantial reform for the health system in NSW, with Local Health Districts (LHD), Specialty Health Networks (SHN) and pillars all being established, and governance of health service delivery devolved from the Department to providers. The shift from a locally delivered to a centralised training model required evaluation.
THE OBJECTIVE
ARTD conducted an independent review of the District HETI Model to identify what if any changes were needed to improve the overall approach of the delivery of high-quality state-wide online education and training. Subsequently, we developed a multi-layered strategy for the monitoring and evaluation of the eLearning modules.
OUR APPROACH
The review collected evidence through interviews and surveys with key District HETI stakeholders. The review team interviewed seven senior staff members of HETI and District HETI, and six Chief Executives and eight Directors of LHDs and SHNs. All Districts/ Networks and pillar Chief Executives were surveyed, with selected questions for their direct response and other questions that could be delegated to a relevant Director. In developing the Strategy, monitoring and evaluation strategy, ARTD reviewed key policy and program documents consulted key stakeholders through interviews and a workshop with 19 HETI staff. We developed a logic model, protocols for monitoring processes and systems, quality indicators, and guidance for decision-making about the conduct of future periodic impact evaluations of selected eLearning modules.
IMPACT
The review identified achievements and opportunities to be realised. It also provided an understanding of District HETI model workforce needs into the future and an understanding of the perceived strength of partnerships. The evaluation strategy is used to monitor and evaluate the eLearning modules.
_______
Evaluation of the NSW Service Plan for People with Eating Disorders
NSW Centre for Eating and Dieting Disorders, 2018
THE CHALLENGE
In September 2013, the Mental Health and Drug and Alcohol Office (MHDAO) of the NSW Ministry of Health, launched the NSW Service Plan for People with Eating Disorders 2013–2018 (NSW Service Plan). This landmark document, produced by the Centre for Eating and Dieting Disorders (CEDD) in partnership with the Ministry, positioned eating disorders as core business for the NSW Health care system. The NSW Government, through the implementation of the Service Plan, supported changes in service delivery to support access to treatment and improve care for people with eating disorders, no matter where they
live in the state. The Service Plan identified the need for prolonged multifaceted integrated care. It required Districts to build capacity to provide assessment and treatment, with pathways to options for more intense and specialist treatment to be established.
THE OBJECTIVE
ARTD conducted an independent process evaluation of the implementation of the NSW
Service Plan.
OUR APPROACH
The evaluation focus was on the progress being made in the implementation of the Service Plan, not on outcomes. It used a mixed methods design drawing on existing data (performance monitoring, reports and patient data) and collecting new qualitative and quantitative data via surveys and semi-structured interviews with Service Plan stakeholders. Stakeholders include MHDAO, Steering Committee Chair, CEDD team, Local Eating Disorder Coordinators, clinicians who had completed clinical training packages, Local Health District (LHD) Mental Health Directors, LHD Medical Leads and a small sample of people
living with an eating disorder. There was consistency in findings across different data
sources, giving confidence in the findings.
IMPACT
The evaluation found that substantive progress had been made in implementing the NSW Service Plan, shifting the NSW Government’s health system response to eating disorders from ad hoc to a state-wide response. The Service Plan had two companion documents to operationalise its vision for change: The Implementation Plan, 2014-2018 and the NSW Workforce Development Plan for Eating Disorders (2016-2018). The Implementation Plan adopted a philosophy of locally-led services and workforce development, with state-level support.
Although a great deal of work had been done and much achieved across the State, the evaluation found over half of Districts were at an early stage of implementation of their Local Plans with considerable work to be done in establishing models of care, protocols for working across teams and clinical pathways.
The evaluation found that to achieve the Ministry expectation that eating disorder services will transition to become business as usual across NSW Health services during the next five years, that the work being done must be developed and sustained, with support for Districts to implement their Local Plans. Recommendations were made to support the next stage.
More on health sector evaluations: Watch our videos on challenges in health sector evaluations.
