Snapshot evaluation of NSW Health Management Development Programs

NSW Health is an employer for more than 140,000 people in NSW. Developing and enhancing leadership and management skills continues to be a key strategic priority, ensuring that NSW Health employees who are responsible for the management of people and programs have the appropriate skills and knowledge to be successful in these roles.

The Health Education and Training Institute (HETI) has a critical role in achieving these strategic priorities, providing a range of management development offerings that increase the knowledge and skills of NSW Health employees throughout their career trajectories.

ARTD worked with the Health Education and Training Institute to provide a high-level, snapshot evaluation of four major education products that deliver leadership and management development programs to all NSW Health staff.

The evaluation was designed to maximise the value of the extensive existing data on the education products, to better understand their strengths and weaknesses and inform future decisions about delivery.

HETI had extensive participant feedback data, collected via survey across four years of delivery of the programs, containing both quantitative and qualitative responses.

ARTD undertook data analysis of the quantitative feedback data on each program, and also looked at insights emerging across the programs. To do this, we utilised R and the RStudio statistical software, which allowed us to develop code that rapidly produced crosstabulations of each of the various measures of outcomes against different cohort profiles (for example, organisation types or professional groupings).

We partnered with Altometer Business Intelligence to provide qualitative analysis of free text comments using machine learning. Insights were synthesised to help inform and explain the quantitative data findings – particularly where we found difference between groups.

Using a combination of machine learning, statistical analysis code developed in R and dedicated teams, we were able to deliver extensive and useful findings in a very compressed timeframe of less than six weeks from start to finish.

We were able to demonstrate the impact of the programs as reported by participants, backed up with both qualitative analysis and descriptive statistics supported by statistical significance testing. We were further able to show clearly which cohorts across a range of demographic profiles were engaged differently providing useful insights for targeting promotion and consideration when making changes.

The report has been used to inform funding discussions and quality improvement processes for the educational products.