Evaluation Governance: Working with an advisory group to add value
Evaluators are often hired because of their knowledge of evaluation theory, design and program implementation. Or some fall into the role because of their content area knowledge accumulated in another career. Very few of us have the technical expertise or lived experience to plan and implement evaluations in every complex social policy area.
So, it makes sense to bring in the experts. But working effectively with an evaluation advisory group takes practice and preparation. We put our heads together to draw out our insights on working effectively with advisory groups from the numerous groups ARTD staff have collectively worked with or set up over the years.
When do you need an evaluation advisory group?
If the program or policy being evaluated requires a detailed understanding of the context (and understanding context is generally important) to be valid and credible, or you need subject matter expertise, you will need to draw on advisors.
This is particularly true for evaluations of policies and programs focused on outcomes for Aboriginal and Torres Strait Islander people. (It’s a strong recommendation of the Ngaa-bin-yaa Evaluation Framework to use an evaluation advisory group.) An Aboriginal advisory group not only supports a strong and useful evaluation, but also Aboriginal ownership and knowledge building. As ARTD’s Director of Aboriginal Partnerships and Programs, Simon Jordan notes, an Aboriginal advisory group can ensure the evaluation is culturally appropriate and sensitive.
“The thing about having a really good Aboriginal reference group is that they can help prepare the space for you culturally, so it’s not just you doing the work. They can provide insights into contentious issues or sensitivities in the communities you are going to be working in, and are able to provide different interpretations on what is considered a ‘good’ outcome. Having them thoroughly involved is a way to have the voice of community brought to bear on big, mainstream evaluations.” – Simon Jordan, ARTD Director – Aboriginal Partnerships and Projects
Advisory groups can also be important when your work is about programs or interventions targeted to people with disability or experiences of mental and physical ill health, or a lived experience of other social issues, such as homelessness or unemployment.
Some states and territories provide guidance on when to set up advisory groups. For example, the NSW Evaluation Guidelines recommend setting up an advisory group for projects that are of high strategic priority to government, being sure to include representatives from all participating agencies.
What does an advisory group do?
What the advisors do is up to you and the terms of reference.
The purpose of an advisory group may be to strengthen the evaluation team’s understanding of the context for an evaluation, to advise on stakeholder engagement, to ensure methods are both feasible and robust and to help interpret findings, depending on the terms of reference. As Bianca Jarrett, ARTD’s Manager of Aboriginal Partnerships and Projects notes, “They can benefit the project by improving collaboration, advocating on behalf of the project, providing advice, and representing the views and work of their particular area.”
Who should be involved?
Ensuring you have the ‘right people’ in the room is crucial.
When choosing advisors, consider whether there are knowledge gaps you need to fill, or particular experiences you want to capture.
Being mindful of how many people are involved is also important. Too few advisors might mean a narrow perspective, but too many advisors may make it hard for people to be heard. As ARTD Analyst Holly Kovac notes, “You can’t have a group that’s too big and then expect everyone’s going to feel their opinion is heard and validated. At a certain size it begins to feel tokenistic.”
How do you get the best out of your group?
It’s important to plan for advisors’ input, ensuring each member’s experience is valued. This will ensure better outcomes both for the evaluation and for the members themselves.
You can give advisors certainty and keep your meetings focussed by developing clear Terms of Reference. This helps people understand their responsibilities as members, and how these link to the broader project purpose.
With a diversity of voices in the room, evaluators must take the time to build rapport and trust between members. Beyond the Terms of Reference, you might want to set out some ground rules for the conversation, such as equal voice and the confidentiality of the room.
If your project needs advisors with lived experience and technical expertise, you might want to consider two separate advisory groups, or if a single group, to consider the balance of numbers and power dynamics.
“Sometimes it’s better not to mix those groups. It can sometimes be very effective to bring everyone together, but sometimes you need to consider if community members will feel less confident to contribute in a group setting,” says Bianca.
The Chair’s skillset is fundamental to cultivating an environment of trust. Ideally, the meeting Chair should be a skilled facilitator, who can support productive conversations even amongst members who disagree. The Chair can keep the conversation on track—opening it up when the issues dictate exploration; and moving it along to ensure the agenda is met.
When should the group meet?
It’s important to bring advisory groups together on a regular schedule that aligns to key project milestones. Too many meetings can be overwhelming and inefficient; too few can be a missed opportunity to hear key insights.
Respect your advisors’ time by being prepared. Ensure you have distributed the agenda, meeting minutes and background documents well in advance of meetings to give your advisors time to process and prepare their best insight and advice.
What are the benefits?
Done well, involving advisors or advisory groups in an evaluation is mutually beneficial.
On the one hand, advisors’ insights help evaluators clearly understand and situate findings within a broader context. Discussions with advisors offer us a different ‘lens’ through which we can reflect our practice and findings, as well as providing opportunities for shared decision making and learning, which can enrich the project.
“We used a reference group of eight people, who had used an Ability Links program. With the assistance of two language interpreters, a physically accessible venue and a discussion approach that was inclusive of people with a vision impairment, we talked with the reference group about how the program was working, and how it could be improved. This discussion ensured our team could interpret the findings in context.”– ARTD Partner & Managing Director Jade Maloney
(read more about this project)
On the other hand, advisory groups can be an opportunity for members to share their expertise and insights with other key stakeholders, and an opportunity to learn about the evaluation profession, or the processes for developing policy and programs.
Further reading
- https://www.betterevaluation.org/en/evaluation-options/advisory_group
- Centers for Disease Control and Prevention. Division of Nutrition, Physical Activity, and Obesity. Atlanta, GA (2011). Developing and Using an Evaluation Consultation Group. Atlanta, GA.
