Oct

2021

Listen up evaluators!

News

We talk a lot in our work about the power of evaluative thinking, and much of our work is about listening. But while we put the two together intuitively, we perhaps talk less about the power of ‘evaluative listening’ as a critical skill of an evaluator, and the value that skillset brings to the design of better systems, projects and policies.

‘Evaluative listening’ may be used in contexts other than evaluation to refer to critical listening in order to make a judgement or decision. However, an evaluator is bridging several listening styles and levels when doing interviews, focus groups or stakeholder consultations, and bringing them all together using evaluative thinking to provide a picture of context, experiences and experiential knowledge.

Listening to build and maintain client relationships and deliver useful evaluations

In evaluation, listening happens right from the very beginning. In order to build the kind of relationship required for evaluators to be able to act as the ‘critical friend’, and to allow program managers and policy makers to receive recommendations, we have to first provide evidence that we understand the needs, context and players involved.

It is through the process of listening to clients that we develop our relationships, demonstrate we understand the sticky parts of their work, validate their efforts and then earn the right as consultants to provide constructive critique.

Being mindful while listening

“Preparation for listening consists of getting your body and mind in tune for the listening experience. It means that you think about setting aside your personal opinions and get yourself focused on understanding the person you’ll be interviewing.” [1]

Being mindful of what you as a facilitator or interviewer bring is absolutely critical as an evaluator. Oscar Trimboli, host of the Deep Listening podcast, calls this Level 1: Listening to yourself.

Evaluators must be able to be mindful of themselves within conversations, how much they interrupt, or whether their mind races ahead to fill in the blanks, how comfortable they are with leaving silences and allowing the interviewee to fill them, what their mind is doing (whether it is wandering), and whether they are listening from a biased or judgemental perspective. They also need to make sure they are listening, rather than formulating their own response. This can be particularly difficult if what you are talking about has pressed on a ‘hot button,’ as Dr Richard Krueger says:

“People have hot buttons they’re not even aware of until someone pushes them—something is said and you feel yourself having an emotional reaction… Things could be said in a group that bother you because they conflict with your values, or your judgment, or something that’s troubling to you. If you’re aware that that might happen that’s the key thing. And if it happens to you, you must have the discipline to control your behavior.” [1]

Reading the relationship

Anna Sale, the host of the podcast sensation Death, Sex and Money (a podcast about the big questions and hard choices that are often left out of polite conversation) talks about treating an interview like a relationship.

“As you talk, you need to pay attention to how you are relating to your conversation partner, even as you press on uncomfortable subjects. Notice their body language, or when their answers become short or clipped, and what makes them expand further. You are watching for cues about where you can push more, and what needs to be left for another time. This is the part of the conversation that is not about what you are hearing, but how you are hearing it. By paying close attention, you are tending to your relationship, the emotional dynamic between you, as you exchange words and ideas. And you can learn a lot about acknowledging what you’re noticing about them in the conversation.” [2]

The skill of reading the relationship using the non-verbal as much as the verbal, is also tied to your ability to be mindful and listen to yourself.

Empathetic listening

Empathetic listening requires openness and receptivity to what the speaker is trying to get across, with curiosity, and without judgement. It’s a difficult skill to master, but is vital to ensure evaluators can conduct research, particularly with vulnerable groups, without doing harm. Done well, it reduces defensiveness, supports people to feel heard, encourages people to trust and open up, and can help orient people towards solving problems or developing solutions.

Listening to and drawing out the differences

While humans like things to fit neat categories, it’s not always desirable to reduce the complexity. When listening to multiple perspectives across qualitative research like interviews or focus groups, there may be an impulse towards seeking the commonalities. However, as Luke Craven writes,

“One of the basic principles of complexity is that different individuals experience complex systems in different ways. Some of those differences are probably superficial, but there are others that will be material, incongruous and paradoxical. The common impulse is to iron over those differences (what I’ve seen called “analysis” or “synthesis” or “triangulation”) in search of shared understanding, but we should not be afraid of difference and multiplicity in our work. Accommodating it does more justice to complexity than pretending it does not exist.”
(Luke Craven, Pig on Tracks, ‘There is no elephant’ #27)

It’s often the interviewer and facilitator’s job to listen for ‘sticky points’, friction, power imbalances, and differences, and draw them out into the light for discussion (and sometimes in the process creating greater shared understanding).  It’s not an intuitive thing to do, given most people are trained that it is impolite to have difficult conversations, but there is so much value in doing it (this is the reason that despite the three difficult topics of the podcast Death, Sex and Money, so many people tune in year after year).

Listening when face to face isn’t an option

Listening online is different and often more challenging than listening in person – it’s more difficult to get turn taking right, resulting in talking over one another – which diminishes the feeling of being heard – and it makes it very difficult to pick up on non-verbal cues that can provide context for what a person is feeling as they are talking.

“Authentic expressions of emotion are an intricate array of minute muscle contractions, particularly around the eyes and mouth, often subconsciously perceived, and essential to our understanding of one another. But those telling twitches all but disappear on pixelated video or, worse, are frozen, smoothed over or delayed to preserve bandwidth…Video chats have also been shown to inhibit trust because we can’t look one another in the eye.“ [3]

The knowledge that the technology can cut out at any time also makes silences more difficult to read, and any deeper communication is at risk of interruption at any time by technology [4]. All of this makes rapport more difficult to build, which means the interviewee or focus group member may not be as willing to share certain types of information.

Webinar technology has also made us all much more aware of the image we are ‘creating’ when online. We recently had a lunchtime learning on constructionism in evaluation. Instead of facts, constructionists believe you can find ‘narratives’ or versions of people’s selves which are contextually based and socially produced. So the ‘self’ an interviewee is creating in the context of an interview, is shaped intentionally by what the interviewee wants the interviewer to see, but also by the social construct of the ‘interview’ (and their expectations of how an interview should go), and their interaction with the interviewer. We humans (particularly those whose childhoods featured ready access to the internet) have been trained to self-represent or even ‘perform’ ourselves online in different ways than we might in a face to face environment (for more on this, read Trick Mirror by Jia Tolentino). We have also learned, rather more slowly, that what is on the internet stays on the internet. For those of us who are more privacy-oriented, this is likely to consciously and subconsciously orient the way we interact online, regardless of whether recordings of our participation are likely to be made public, or not. All of these things must be part of an evaluator’s awareness when listening in an online environment.

In certain circumstances, all of these considerations may mean phone interviewing is more suitable, or it might mean using shared activities to build trust, or doing some of the work online and some offline. ARTD Director Simon Jordan changed some of his approaches in the early days of COVID-19 to incorporate some of these different types of engagement:

“To conduct our work, we’ve had to be more innovative and interactive in the tools that we use: we’ve used facetime, things like interactive online whiteboards and online vision cards…We’ve broken down our workshops into more digestible chunks, and have given participants homework (like asking people to complete SWOTS and survey), so they are bringing even more value back to the next workshop we run with them… I think there would have been an enormous challenge in using these technologies however, if there were no existing relationships between us and the service providers we work with, and their communities. When using online tools, we needed to be more aware to check in with people, be more aware of facial cues, and drawing out participants who tend to be more silent – even more than in face to face workshops.” (Read the full blog)

When to stop listening and start analysing

While it is important that evaluators are able to listen to and capture a wide breadth of experiences in qualitative research, it’s also vital that we are able to identify when to stop listening and start analysing.

Finding the saturation point of qualitative data is a judgement guided by time and budget constraints, but also by applying evaluative thinking throughout the process of data collection, and listening for patterns and repetition across the data, and noticing the point at which there are ‘diminishing returns’ from conducting further qualitative data collection.

Reflecting on listening

Reflective practice is one of the key competencies of evaluators. Thinking critically about data is what we do as a job, but thinking critically about our own practice and approaches is the process by which we are able to do the work in a valid way. Reflective practice can be done alone, or together – frequently we have lunchtime learning sessions focussed just on reflective practice – and is about making space to ask oneself questions like ‘what happened’, ‘why’ and ‘what are the benefits of changing how I approach things or do things’ and ‘what will I learn or change for next time?’[5]. It’s so important that one evaluator has made a colouring book to help evaluators do reflection!

As Michael Quinn Patton says:

“Reflexivity reminds the evaluator to be attentive to and conscious of the cultural, political, social, economic, linguistic, and ideological origins of one’s own perspective as well as the perspective and voices of those you gather data from and those to whom you present findings” [5]

I listen to the podcasts of psychotherapist and relationships expert Ester Perel where she records one-time therapy sessions between two people. I am often struck by how she reflects on her own process as psychotherapist as part of the podcast. She will occasionally stop herself, comment that she has miscommunicated and correct herself. She sometimes edits in narration reflecting on what she could have said or done better in a session – including when she has missed key moments in the conversation where it was appropriate to dive deeper into a problem because her attention was elsewhere, or where she could have left more silence before jumping in to say something. It’s a fascinating and delightful opportunity to observe an expert reflective practitioner, and a helpful tool for evaluators to observe to grow their own reflexive muscles.

References