ARTD News
May 2013
Findings on ADHC autism programs published
Ageing Disability and Home Care (ADHC) has released the high-level findings from ARTD’s evaluation of four models of early intervention for children with autism aged 0 to 5 years and their families. The four programs—Footprints (Autism Behavioural Intervention NSW), Building Blocks, More Than Words and Autism Pro (Aspect NSW)—were funded through Stronger Together: A new direction for disability services in New South Wales, 2006–2016. All of the programs were designed in line with best practice principles, but they varied in terms of their philosophical orientation, delivery (mode, intensity, duration) and the intervention focus (child or parent).
The evaluation compared the outcomes for children across the four programs, as well as the cost-effectiveness of the programs, drawing on quantitative and qualitative data. The findings showed that children who took part in one or more of the four programs—within the context of receiving other services, supports and therapies through other sources, including the Helping Children with Autism Package—demonstrated improved skills, abilities and behaviours. Their parents’ knowledge, understanding and ability to cope also increased. The evaluation will inform ADHC’s decisions about the future direction of programs to support children with autism and their families.
This work builds on our other work in the autism area, in particular the evaluation of the Helping Children with Autism Package for the Commonwealth Department of Families, Community Services and Indigenous Affairs, and adds to the evidence base about effective supports for children with autism and their families.
ARTD Sponsors IPAA Collaboration Award
ARTD is continuing to champion collaboration in government by sponsoring the NSW Institute of Public Administration Australia’s (IPAA) Collaboration Award again this year. The Collaboration Award recognises individuals who
– have established effective joint working arrangements that bring together colleagues from multiple teams, departments or other organisations
– can clearly identify and communicate the benefits of a collaborative approach
– have delivered improved, coordinated and efficient services.
Our Principal Consultant Sue Leahy presented the award to the 2013 winner, Dion Peita, at a special reception after IPAA’s Special Forum: Reforming to Create Value on Wednesday 8 May. Dion was recognised for his innovative work leading a collaboration between the Australian Museum and the Fairfield Office of Juvenile Justice NSW.
Concerned with the over-representation of Pacific youth in the NSW criminal justice system, in 2008, Dion began exploring the potential for the Australian Museum to help at-risk young people stay out of jail. Juvenile offenders of Pacific background now have the chance to access the Museum’s cultural collections as part of special programs. This has been achieved through collaboration between the Museum and organisations that provide services to at-risk young people. As a result of Dion’s vision, hundreds of young people have gained awareness of their cultural background through handling cultural artefacts and engaging with cultural experts, discussing issues around cultural identity and artistic expression.
March 2013
ARTD’s Indigenous cadet completes summer internship
Natalie Ironfield recently completed a three-month cadetship with ARTD through the CareerTrackers Indigenous Internship Program. Our consultants took her through the business of evaluation and research from project design to data collection and analysis. Michael Combs, Founder and CEO, thanked ARTD for sponsoring the internship and said, ‘It is vital that we continue to break down barriers and encourage reconciliation in Australia and I believe professional employment is one key element in realising this goal.’ Natalie has now returned to the Australian National University in Canberra, where she is completing her studies in International Relations. We hope to continue our relationship with Natalie in the future and wish her well with her studies this year.
Evaluation of the Australian Volunteers for International Development (AVID) program
AVID places Australian volunteers with organisations in 37 countries as part of the Australian Government’s foreign aid contribution. Growing volunteer numbers and a recent restructure of the AVID program in AusAID have presented an opportune time for an evaluation, and in 2012 the Office of Development Effectiveness (ODE), AusAID, commissioned ARTD to evaluate AVID to inform future directions. The evaluation, led by Andrew Hawkins, started with a review of international peer-reviewed and grey literature, recently published on the ODE website. Because the scope of the AVID program is so large, ARTD’s work focused on how the program was working in three case study countries: the Solomon Islands, Vietnam and Cambodia. Andrew, with Emily Verstege, Ofir Thaler and ODE representatives, held more than 120 interviews with host organisations, volunteers and AusAID staff during field visits to the case study countries. Other methods included analyses of all articles about AVID published in the Australian media in 2011–12, mapping volunteer assignments in the case study countries against AusAID priorities, analysis of end-of-assignment reports filled out by host organisations and volunteers, and a survey of host organisations. It also drew on a survey of returned volunteers (2006 to 2011) carried out by ORIMA Research. ARTD’s evaluation report will be finalised by June 2013.
We have started the Evaluation of the National Partnership Agreement on Preventive Health
The Australian National Preventive Health Agency (ANPHA) has commissioned ARTD Consultants, in partnership with the Prevention Research Collaboration, University of Sydney, to undertake this large two-and–a-half year national evaluation. The National Partnership Agreement on Preventive Health (NPAPH) is an initiative of the Council of Australian Governments (COAG) and the largest ever national investment in preventive health—it aims to achieve better health for all Australians. The National Evaluation of the NPAPH will be conducted in several stages from January 2013 to June 2015. The evaluation will focus on assessing the benefits of the partnership approach for delivering preventive health initiatives across the country and review evidence of the impact of the NPAPH in all jurisdictions.
December 2012
Evaluating capacity building and organisational change for achieving health improvement goals
Principal Consultant Dr Margaret Thomas recently presented a paper at the European Conference of the International Union for Health Promotion and Education, held in Tallinn, Estonia. Margaret’s paper offered a theory-based approach to evaluating capacity building and organisational change. For her presentation she drew on ARTD evaluation projects where there was a focus on assessing the outcomes of capacity building and organisational change efforts for health improvement goals. For more information about the conference go to: http://www.conferences.ee/iuheli2012/index.html.
Advice on State of the Public Sector in NSW report
The Public Sector Employment and Management Act (PSEMA) requires the Public Service Commissioner to present a report on the state of the public sector in NSW to the Premier each year. While other Australian and international jurisdictions have been producing these kinds of reports for some time as an accountability measure, the NSW Public Service Commission (PSC) as a new and independent body, wanted to use their reports as a communication tool and to position the NSW PSC to best achieve their aims in a changing public sector context. Over six weeks, ARTD and our partners (academics from Monash University and visual communications specialists from Equation), worked with the PSC to define the strategic directions for State of the Public Sector reporting in NSW to meet legislative and other objectives and then define the message, structure, content and design of the 2012 report.
ARTD provided the NSW PSC with a work plan that reflected the tight time frames for the 2012 report. The work plan also balanced the objectives of achieving best practice reporting whilst minimising the burden of data collection for agencies and staff.
The report was released in November 2012, titled How it is: state of the NSW public sector report 2012
HCWA Package evaluation findings at the PsychDD conference
ARTD evaluated the Department of Families, Housing, Community Services and Indigenous Affairs’ (FaHCSIA) components of the Commonwealth Government’s Helping Children with Autism Package over three years from 2009–2012. The evaluation findings about the Package, which centred on a new mechanism for funding early intervention services for children with autism—$12,000 of funding for families to use with services on an approved panel of providers before the time their child turns seven—provided some important learnings for individualised funding approaches for children with disability and for supporting families. Given this, ARTD and FaHCSIA were asked to present at the PsychDD conference on November 30 in Sydney. The presentation covered the evaluation findings and how FaHCSIA had responded to the suggested refinements.
The summary and full reports on the evaluation findings are available of the FahCSIA website http://www.fahcsia.gov.au/our-responsibilities/disability-and-carers/program-services/for-people-with-disability/helping-children-with-autism.
October 2012
Community Builders: first monitoring reports complete
ARTD has recently completed the first monitoring reports for the NSW Department of Family and Community Services’ Community Builders program. This program funds community organisations for activities that build community strength and capacity.
Using the monitoring system we developed and piloted with services, we have produced state, regional and individual service level reports.
The state and regional level reports provide an overview of the program’s scope, reach and outcomes. They are an important tool for funders to understand how the program is working and to inform decision making.
The individual reports—all 476 of them—provide each service with data on their service delivery and feedback from clients, along with average scores on performance measures for the region and the state. These reports are a useful resource for services to understand how they are travelling and to consider further improvements to services to reduce community inequality and disadvantage.
Connecting to Country evaluation
Connecting to Country is a new and innovative program for teachers and principals in New South Wales funded by the Commonwealth Government under the Council of Australian Governments’ (COAG) Closing the Gap strategy. It focuses on developing teachers’ and principals’ willingness and ability to establish relationships with Aboriginal students through learning about their cultural, linguistic and family backgrounds, and then using that knowledge to inform classroom practice and pedagogy.
The program begins with a three-day cultural immersion workshop—Being Culturally Aware, Becoming Culturally Inclusive: A Pathway to Cultural Competence—developed by the NSW Aboriginal Education Consultative Group (AECG) and implemented by Regional and Local AECGs. Following this, participants attend a two-day professional learning workshop to build on and strengthen their capacity to plan, develop and implement culturally inclusive programs and school leadership practices.
The NSW Department of Education and Communities has engaged ARTD to evaluate the program’s effectiveness, strengths and challenges and the impact it has had on principals and teachers, Aboriginal students, their families and communities. The evaluation will contribute evidence for the future sustainability of the Program. The evaluation is due to report in March 2013 and the findings will be made available to interested stakeholders including the Commonwealth Department of Education, Employment and Workplace Relations, the NSW Aboriginal Consultative Group Incorporated, the NSW Teachers Federation, the Primary Principals’ Association and the Secondary Principals’ Council.
August 2012
Evaluation of Lifeline Online Crisis Support Chat
In 2011, Lifeline trialled an online crisis support chat service that aimed to replicate Lifeline’s 13 11 44 telephone crisis service in an online environment. We worked with Lifeline to evaluate the trial at the three sites using data from the online counselling system, analysis of chat transcripts and interviews and focus groups with crisis workers. Lifeline used the results of the evaluation to further develop the Crisis Support Chat Service. And the service was recently launched as a permanent, nationwide service, announced in the Sydney Morning Herald.
Pro bono work for Kool Kids
The Kool Kids Club, delivered by Weave, runs free after school and holiday and activity programs in La Perouse and surrounding areas. Two of our consultants supported the Club by developing a program logic describing the outcomes for children, their families and community and evaluating the activities. The evaluation found the club is delivering an age- and culturally-appropriate program of activities consistent with the program logic and it is valued by children, families and the local community We are now mentoring staff to develop and implement tools to collect data on outcomes so the Club can monitor its own work.
Andrew Hawkins’ presentation on evaluating online services now available online
ARTD Senior Consultant Andrew Hawkins’ presentation—an introduction to evaluating websites and online services—to the NSW Australasian Evaluation Society (AES) meeting in Sydney in May 2012—is now available online. Those who missed out but are keen to learn more about the basic concepts for evaluating websites— search engine optimisation, accessibility, use and useability—and relevant links and resources can find out more here http://www.youtube.com/watch?v=hS6UUefK1LU.
ARTD a key contributor to the Australasian Evaluation Society 2012 conference
We recognise the importance of keeping up to date with evolving research methods and collaborating with our professional colleagues to share learnings. For this reason, ARTD is again a proud sponsor of this year’s Australasian Evaluation Society Conference, which will be held in Adelaide on 27–31 August.
This year’s conference is about Evaluation in a Changing World. Three of our Consultants will be grappling with the implications of the implications of an evolving context for our evaluation approaches and methods, presenting papers at the conference.
Senior Consultant, Andrew Hawkins will present Evaluation and Government 2.0.
One of the most obvious ways in which the world is changing is the degree to which the world’s information is being created, organised and disseminated using the internet. Government sponsored services are also increasingly delivered over the internet. As evaluators are information workers trying to understand government services it follows that evaluators should have some understanding of the internet and what makes for an effective online service. Andrew will discuss what you would need to know if you were asked to evaluate a service delivered partially or wholly over the internet.
Senior Consultant, Emily Verstege will present Contextualising change: How big is big?
By synthesising findings from other autism-specific research and evaluation studies, this paper contributes to the rapidly changing, diverse literature on the effectiveness of early intervention programs for children with autism spectrum disorders and their families. It also proposes a methodology that is useful for evaluators to consider when seeking to measure and interpret change as the result of a program or policy.
Consultant, Narelle Ong will present Health service policy uptake: evaluating and influencing change through an action research approach
The presentation will describe an evaluation of a complex mental health policy, which is part of a high-profile national initiative (the National Perinatal Depression Initiative). Narelle will talk about the ARTD team’s experience of using an action research approach to influence change across a large and complex system of approximately two hundred health services.
To find out more about the conference visit the AES 2012 conference web site: www.aes2012.com
ARTD sponsors IPAA Collaboration Award
Collaboration is increasingly a key part of effective government responses to addressing complex issues, like homelessness for example. ARTD is delighted to be sponsoring the NSW branch of the Institute of Public Administration Australia’s (IPAA) 2012 Collaboration Award. The Collaboration Award recognises individuals who
– have established effective joint working arrangements that bring together colleagues from multiple teams, departments or other organisations
– can clearly identify and communicate the benefits of a collaborative approach
– have delivered improved, coordinated and efficient services.
Because we know how important collaboration can be to effective service responses in a range of areas, we are proud to be involved in recognising the work of public sector professionals who have achieved excellence in collaboration.
The awards will be presented at a cocktail reception after the IPAA NSW State Conference on Thursday 9 August.
June 2012
Evaluation of the U-Turns for Youth program
Over the last year ARTD has supported Bankstown City Council with their evaluation of the 'U-Turns for Youth' program. This program works with local youth who are at risk of disengaging from school or getting involved in motor vehicle crime, providing them with automotive and life skills workshops to support their engagement in education, training and employment and access to youth support services. ARTD began by working with the project manager to develop an evaluation framework and plan for the ‘U-Turns for Youth’ project. This included data collection methods and tools the Council could use to collect sufficient data for a program evaluation. The Council then commissioned ARTD to analyse the data they collected and write the final evaluation report in June 2011. The report will be used to support and assist other local government areas to implement similar initiatives.
Margaret Southwell of Bankstown Council had the following to say about working with ARTD:
“When we developed the idea of running the ‘U-Turns for Youth’ program across Bankstown and Canterbury we knew we needed to collect a broad range of data to assess the impact of the program. ARTD were able to work within our budget and time constraints to guide us in the development of appropriate data collection tools and then mentor our project manager through the process of collecting the data. The final evaluation report was professional, in depth and delivered on time. I look forward to working with ARTD again!”
ARTD Review of Respite Support for Carers of Young People with Severe or Profound Disability
ARTD recently evaluated FaHCSIA’s Respite Support for Carers of Young People with Severe or Profound Disability. The program provides immediate and short-term respite to carers of young people with severe or profound disability and facilitates access to information, respite care and other support or assistance. The evaluation found is well targeted and is making a real difference in the lives of carers and their families. Senator Jan McLucas the findings of this evaluation will help inform the design and development work needed for the National Disability Insurance Scheme. For more see this link.
April 2012
Evaluating websites and online services
The internet is increasingly being used to support and deliver social policy and programs by governments around Australia and internationally. Our Senior Consultant Andrew Hawkins has managed several evaluations of online services, including evaluations of Gambling Help Online (Victorian Department of Justice), Lifeline Online Crisis Support Chat Trial (Lifeline Australia) and the Sage Centre (NSW, Families and Community Services).
Do you need to commission an evaluation of an online service? Have you been asked to deliver one? Would you know which questions to ask to effectively assess the online service and how to answer them?
Following his presentation on evaluating online services at the American Evaluation Association (AEA) annual conference in Los Angeles, California, in November 2011 and his webinar for the AEA in March 2012, Andrew is presenting an introduction to evaluating websites and online services at the NSW Australasian Evaluation Society (AES) meeting in Sydney on 24 May 2012.
The presentation will address the four basic components of a successful website or online service: search engine optimisation, accessibility, use and useability. The demonstration will explain basic concepts, showcase tools, and provide links and resources for those wishing to commission, or try their hand at, evaluating a website.
For more information about Andrew’s presentation, please contact Ben Barnes at the AES on 0423 208 676 or benbarnes@optusnet.com.au
February 2012
ARTD review of the School Based Management Pilot published
Client: Department of Education and Communities
Timing: August 2011–October 2011
The pilot
Forty seven schools of various types and sizes and located across NSW have been involved in the pilot over two years, concluding at the end of 2011. The School Based Management Pilot allows schools increased flexibility and authority in decisions about their management, including human, material and financial resources. All 47 schools in the pilot are engaged in managing both staff and financial resources in new ways.
The Independent Review
ARTD Consultants was selected to undertake an Independent Public Review of the pilot in August 2011. The review encompassed the 47 schools in the pilot and particularly aimed to obtain the views of principals and to capture their experiences of the pilot. Based on the terms of reference, the review concentrated on four key areas:
1. Responsibility, authority and decision making
2. Information and systems
3. Risk management and accountability
4. Capacity, capabilities, cultural and organisational change.
ARTD Consultants collected primary data through interviews with members of the School Pilot Oversight Group, a survey of pilot school principals and interviews with 19 principals and a small number of Parents and Citizens association (P&C) representatives. We reviewed data and information available from the internal evaluation of the pilot, conducted a focus group with five Department staff and analysed State Office data on pilot initiatives.
Publication of findings
The findings, conclusions and recommendations of the Independent Review have been published in a full and summary report available on the Department of Education and Communities website.
The full report: https://www.det.nsw.edu.au/media/downloads/about-us/statistics-and-research/key-statistics-and-reports/irsb-management-pilot.pdf
The summary report: https://www.det.nsw.edu.au/media/downloads/about-us/statistics-and-research/key-statistics-and-reports/irsb-management-pilot-sr.pdf
ARTD developing the evaluation framework for the NPAPH
ARTD has recently been awarded the contract to develop an Evaluation Framework for the National Partnership Agreement on Preventive Health (NPAPH) for the Australian National Preventive Health Agency (ANPHA) (www.health.gov.au/internet/main/publishing.nsf/Content/phd-prevention-np).
We will be working in partnership with colleagues from the Prevention Research Collaboration, University of Sydney (http://sydney.edu.au/medicine/public-health/prevention-research/). The project will include consulting with government preventive health departments and non-government organisations around Australia.
ARTD offering Indigenous cadetships
Are you
- an Indigenous person interested in a career in research, evaluation or strategic consulting
- interested in getting paid while you study a diploma, associate diploma or degree?
ARTD Consultants is offering two cadetship positions for Indigenous people studying full-time in 2012.
We’re looking for people studying
- Aboriginal and Australian studies
- Arts
- Community development
- Computing science
- Economics
- Environmental studies
- Information technology
- Management studies
- Program and policy development
- Project management
- Research and evaluation
- Social science.
The cadetship offers
- paid work placement of 12 weeks for each year of your cadetship
- professional guidance and mentorship through your study and work placement
- financial assistance
- opportunity for ongoing employment after successful completion.
This program is ideal if you
- are eager to learn and enjoy working as part of a team
- are organised, flexible and good at managing your time
- have good written and oral and communication skills
- have either an understanding of research methodologies and practices and/or skills and interest in web-based and other data system development
- are interested in a career in applied policy, research, evaluation or consulting.
How to apply
Refer to the Indigenous cadetship handbook for the selection criteria and application form, which can be found at http://www.det.nt.gov.au/__data/assets/pdf_file/0018/4923/NICPhandbookV2.pdf For more information see https://www.ics.deewr.gov.au/
For more details please contact Sue Leahy on 02 9373 9906.
This project is supported by funding from the Commonwealth Government under its Indigenous Cadetship Support, administered by the Department of Education, Employment and Workplace Relations.
November 2011
Evaluation of online services and websites - AEA 2012
Governments are increasingly using the internet to support or deliver social policy and programs and it is important that we as evaluators have the knowledge and tools to effectively evaluate websites or online services. Our Senior Consultant Andrew Hawkins has managed several evaluations of online services, including evaluations of Gambling Help Online (Victorian Department of Justice), Lifeline Online Crisis Support Chat Trial (Lifeline Australia) and the SageCentre (Ageing, Disability and Home Care). Andrew is presenting a paper on the evaluation of websites and online services at the 2011 American Evaluation Association annual conference in Los Angeles, California. His See http://www.eval.org/eval2011/
September 2011
ARTD work with Community Builders program
ARTD have developed an evaluation framework, strategy and monitoring system for the NSW Department of Family and Community Services Community Builders program, funding community organisations for activities that build community strength and capacity. The Community Builders program has been developed in response to an emergent body of evidence suggesting that community strengthening is an effective way of reducing inequality and disadvantage.
The evaluation framework provides a basis for evaluation and monitoring. It is shaped by the program’s aims and objectives, informed by evidence about constitutes a strong community, and structured by program logic for Community Builders. The evaluation strategy aims to generate an evidence base for the program that can inform decisions by stakeholders at different levels. It sets out important strategic issues, key questions for evaluation, and feasible methods to address them.
ARTD have worked in collaboration with Community Services and the Local Community Services Association (LSCA) in developing the framework and monitoring system. Principal Consultant, Sue Leahy recently co-presented with these partners to the 2001 LCSA conference on the approach to and progress with this work.
March 2011
Our work with Camp Quality
ARTD recently worked with Camp Quality on a major national research project to find out more about the needs of children with cancer, their siblings and parents throughout their cancer journey, and how Camp Quality and other cancer support organisations could better meet these needs.
The project involved focus groups with Camp Quality children with cancer, their siblings and parents at 18 camps across Australia, a survey of Camp Quality families, an online survey of relevant health professionals, and a literature review. Camp Quality is using the findings of the research to inform further development of the programs they deliver to support families around Australia. To find out more about Camp Quality see www.campquality.org.au/public/home.aspx.
ARTD published in the Australian Journal of Primary Health
ARTD Consultants Margaret Thomas and Narelle Ong recently worked with the Clinical Education and Training Institute, Rural Division (previously the NSW Institute of Rural Clinical Services and Teaching) to evaluate the Rural Research Capacity Building Program. The NSW Health Framework for Capacity Building—which highlights elements of workforce development, organisational development, resource allocation, leadership and partnership—was used to guide both the development and implementation of the program and the program evaluation. As part of the Institute’s overall evaluation, ARTD interviewed health professionals in the Program, and analysed the data to assess the Program’s effectiveness in improving research capacity among rural health professionals and the impact in their workplaces. The evaluation also identified barriers to research capacity building in rural NSW health services and possible improvements to the Program. The paper they co-authored with our clients, Dr Emma Webster and Ms Linda Cutler, has now been published in the Australian Journal of Primary Health, Volume 17, No 1 2011 p107–113. See http://www.publish.csiro.au/paper/PY10060
ARTD report tabled in Parliament
ARTD completed the five-yearly legislative review of the Medicare Provider Number legislation on behalf of the Department of Health and Ageing in 2010. The review involved extensive consultation with the sector, in particular medical colleges, professional bodies and relevant government officers. The review report was tabled in Federal parliament in February and is available at http://www.health.gov.au/internet/main/publishing.nsf/Content/work-pubs-provleg-10-rpt
Writing at ARTD
We take our writing seriously at ARTD. We know our clients need reports that clearly and concisely answer their key questions. Our staff have trained with Mark Tredinnick, author of The Little Black Book of Business Writing. You can find out more about Mark’s writing and consulting work on his website www.marktredinnick.com.au.
We also have internal processes to ensure the quality of our written documents. ARTD’s Principal Consultants sign off on all reports, and we have an experienced editor on staff.
February 2011
Assessing cost-effectiveness in evaluation
Cost-effectiveness and cost-benefit analysis are an increasingly important part of our work in evaluation. To build on our existing skills (and make sure we’re on top of new methods and issues), one of our Senior Consultants, Andrew Hawkins, recently attended a short course on Advanced Methods of Cost-Effectiveness Analysis run by the Health Economics Research Centre of the Department of Public Health, Oxford University. The course covered both the theoretical concepts and practical exercises in analysing health outcomes data, collecting cost data, discounting costs, dealing with missing data, bootstrapping sample data, constructing decision trees, Markov modelling, and calculating and interpreting incremental cost-effectiveness ratios (ICERs) and the net benefits of health care interventions. You can find out more at www.herc.ox.ac.uk/courses/advancedmethodsmainpage.