Strong Minds Program Evaluation
Western NSW Primary Health Network, 2019
The 2014 National Mental Health Commission Review of Mental Health Programmes and Services highlighted the complexity, fragmentation and inefficiency of the mental health system, and in doing so presented a strong case for reform. The Australian government response recognised the need to implement an ambitious plan for change to reduce the health, social and economic costs of mental illness and suicide for individuals and the community. The new approach to mental health funding includes a focus on person-centred care, with stepped care identified as a priority reform to support redesign of primary mental health care programs and services. This entails moving from a ‘one size fits all’ approach to services matched to individual consumer needs.
The Strong Minds program, funded by Western New South Wales Primary Health Network and delivered by Marathon Health, replaced previous psychological and mental health service models in Western NSW. ARTD was commissioned to evaluate Strong Minds, which aims to provide psychological therapy services for under-serviced groups, including: people living in rural and remote communities; children younger than 12 years; people experiencing, or at risk of, homelessness; women experiencing perinatal depression; and people from culturally and linguistically diverse backgrounds. It also targets people from designated high-needs population groups (such as Aboriginal and Torres Strait Islander people, and people who are at risk of suicide).
ARTD undertook a mixed-methods evaluation to determine whether Strong Minds had met its intended aims and objectives and to inform judgements of value for money. The evaluation drew on a range of existing and newly collected data, including semi-structured interviews with Marathon Health staff, referral partners and other stakeholders, an online survey of Strong Minds clients and general practitioners, and quantitative analysis of Strong Minds referrals data and service and outcomes data for current and former Strong Minds clients.
The evaluation showed the Strong Minds program reduced psychological distress of people with moderate to severe mental health issues, and that there were a range of positive outcomes for participants, including an improved understanding of their mental health, connection with relevant community supports, and re-engagement with family, education or employment.
Strong Minds was shown to complement other supports and services for people with mild to moderate mental health illness within a stepped care model.
The evaluation also highlighted many areas of effective delivery as well as opportunities to improve the program’s implementation and reach.
Snapshot evaluation of NSW Health Management Development Programs
Health Education and Training Institute (HETI), 2020
NSW Health is an employer for more than 140,000 people in NSW. Developing and enhancing leadership and management skills continues to be a key strategic priority, ensuring that NSW Health employees who are responsible for the management of people and programs have the appropriate skills and knowledge to be successful in these roles.
The Health Education and Training Institute (HETI) has a critical role in achieving these strategic priorities, providing a range of management development offerings that increase the knowledge and skills of NSW Health employees throughout their career trajectories.
ARTD worked with the Health Education and Training Institute to provide a high-level, snapshot evaluation of four major education products that deliver leadership and management development programs to all NSW Health staff.
The evaluation was designed to maximise the value of the extensive existing data on the education products, to better understand their strengths and weaknesses and inform future decisions about delivery.
HETI had extensive participant feedback data, collected via survey across four years of delivery of the programs, containing both quantitative and qualitative responses.
ARTD undertook data analysis of the quantitative feedback data on each program, and also looked at insights emerging across the programs. To do this, we utilised R and the RStudio statistical software, which allowed us to develop code that rapidly produced crosstabulations of each of the various measures of outcomes against different cohort profiles (for example, organisation types or professional groupings).
We partnered with Altometer Business Intelligence to provide qualitative analysis of free text comments using machine learning. Insights were synthesised to help inform and explain the quantitative data findings – particularly where we found difference between groups.
Using a combination of machine learning, statistical analysis code developed in R and dedicated teams, we were able to deliver extensive and useful findings in a very compressed timeframe of less than six weeks from start to finish.
We were able to demonstrate the impact of the programs as reported by participants, backed up with both qualitative analysis and descriptive statistics supported by statistical significance testing. We were further able to show clearly which cohorts across a range of demographic profiles were engaged differently providing useful insights for targeting promotion and consideration when making changes.
The report has been used to inform funding discussions and quality improvement processes for the educational products.
Review of District HETI Relationship Model
Health Education and Training Institute, 2017.
District HETI was established in 2012 with a remit for the rapid development of online education and training for the NSW Health workforce. This was a time of substantial reform for the health system in NSW, with Local Health Districts (LHD), Specialty Health Networks (SHN) and pillars all being established, and governance of health service delivery devolved from the Department to providers. The shift from a locally delivered to a centralised training model required evaluation.
ARTD conducted an independent review of the District HETI Model to identify what if any changes were needed to improve the overall approach of the delivery of high-quality state-wide online education and training. Subsequently, we developed a multi-layered strategy for the monitoring and evaluation of the eLearning modules.
The review collected evidence through interviews and surveys with key District HETI stakeholders. The review team interviewed seven senior staff members of HETI and District HETI, and six Chief Executives and eight Directors of LHDs and SHNs. All Districts/ Networks and pillar Chief Executives were surveyed, with selected questions for their direct response and other questions that could be delegated to a relevant Director. In developing the Strategy, monitoring and evaluation strategy, ARTD reviewed key policy and program documents consulted key stakeholders through interviews and a workshop with 19 HETI staff. We developed a logic model, protocols for monitoring processes and systems, quality indicators, and guidance for decision-making about the conduct of future periodic impact evaluations of selected eLearning modules.
The review identified achievements and opportunities to be realised. It also provided an understanding of District HETI model workforce needs into the future and an understanding of the perceived strength of partnerships. The evaluation strategy is used to monitor and evaluate the eLearning modules.
Evaluation of the NSW Service Plan for People with Eating Disorders
NSW Centre for Eating and Dieting Disorders, 2018
In September 2013, the Mental Health and Drug and Alcohol Office (MHDAO) of the NSW Ministry of Health, launched the NSW Service Plan for People with Eating Disorders 2013–2018 (NSW Service Plan). This landmark document, produced by the Centre for Eating and Dieting Disorders (CEDD) in partnership with the Ministry, positioned eating disorders as core business for the NSW Health care system. The NSW Government, through the implementation of the Service Plan, supported changes in service delivery to support access to treatment and improve care for people with eating disorders, no matter where they
live in the state. The Service Plan identified the need for prolonged multifaceted integrated care. It required Districts to build capacity to provide assessment and treatment, with pathways to options for more intense and specialist treatment to be established.
ARTD conducted an independent process evaluation of the implementation of the NSW
The evaluation focus was on the progress being made in the implementation of the Service Plan, not on outcomes. It used a mixed methods design drawing on existing data (performance monitoring, reports and patient data) and collecting new qualitative and quantitative data via surveys and semi-structured interviews with Service Plan stakeholders. Stakeholders include MHDAO, Steering Committee Chair, CEDD team, Local Eating Disorder Coordinators, clinicians who had completed clinical training packages, Local Health District (LHD) Mental Health Directors, LHD Medical Leads and a small sample of people
living with an eating disorder. There was consistency in findings across different data
sources, giving confidence in the findings.
The evaluation found that substantive progress had been made in implementing the NSW Service Plan, shifting the NSW Government’s health system response to eating disorders from ad hoc to a state-wide response. The Service Plan had two companion documents to operationalise its vision for change: The Implementation Plan, 2014-2018 and the NSW Workforce Development Plan for Eating Disorders (2016-2018). The Implementation Plan adopted a philosophy of locally-led services and workforce development, with state-level support.
Although a great deal of work had been done and much achieved across the State, the evaluation found over half of Districts were at an early stage of implementation of their Local Plans with considerable work to be done in establishing models of care, protocols for working across teams and clinical pathways.
The evaluation found that to achieve the Ministry expectation that eating disorder services will transition to become business as usual across NSW Health services during the next five years, that the work being done must be developed and sustained, with support for Districts to implement their Local Plans. Recommendations were made to support the next stage.
More on health sector evaluations: Watch our videos on challenges in health sector evaluations.